RetroSearch Browse

Mon Aug 9 22:21:57 CEST 2004 · http://mail.python.org/pipermail/python-dev/2004-August/047465.html

> This is a question regarding the tokenize.generate_tokens function.
> 
> As it is in CVS HEAD, generate_tokens yields the '@' token as
> having type OP, not type AT. Is this how it should be?

Without looking at the code, I believe tokenize treats all operators
as OP, for a rather liberal definition of "operator".

(A more serious problem that I just encountered in a different
context: when you feed it input containing unsupported characters, the
spaces before those unsupported characters are also returned as
tokens.  I think that's a bug.)

--Guido van Rossum (home page: http://www.python.org/~guido/)

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from http://mail.python.org/pipermail/python-dev/2004-August/047465.html below:

[Python-Dev] tokenize.py (generate_tokens yielding @ as OP)