RetroSearch Browse

Thu Nov 30 22:49:30 CET 2006 · https://mail.python.org/pipermail/python-dev/2006-November/070110.html

On 11/30/06, Fredrik Lundh <fredrik at pythonware.com> wrote:
> Guido van Rossum wrote:
>
> > Are you opposed changing tokenize? If so, why (apart from
> > compatibility)? ISTM that it would be a good thing if it reported
> > everything except horizontal whitespace.
>
> it would be a good thing if it could, optionally, be made to report
> horizontal whitespace as well.

It's remarkably easy to get this out of the existing API; keep track
of the end position returned by the previous call, and if it's
different from the start position returned by the next call, slice the
line text from the column positions, assuming the line numbers are the
same. If the line numbers differ, something has been eating \n tokens;
this shouldn't happen any more with my patch.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://mail.python.org/pipermail/python-dev/2006-November/070110.html below:

[Python-Dev] Small tweak to tokenize.py?