RetroSearch Browse

Sun Aug 8 03:41:56 CEST 2010 · https://mail.python.org/pipermail/python-dev/2010-August/102915.html

I'm trying to add a '?' token to the parser, and weird things
are happening.

I've added a #define to token.h, an entry to _PyParser_TokenNames
in tokenizer.c and case for it in PyToken_OneChar(). But it's
behaving as though the tokenizer is not recognising my token.

I put in some printfs to find out whether PyToken_OneChar() is
recognising it. The results are confusing: while running
pgen, PyToken_OneChar() is being called and recognising the
new token correctly.

However, it doesn't seem to be getting called *at all* when
parsing Python code. I don't see how this can happen, because
pgen seems to use the same tokenizing code as Python itself.

Is there anything else I need to do? Does some file need
to be manually re-made?

-- 
Greg

Home - News ( United States | United Kingdom | Italy | Germany ) - Football scores

Showing content from https://mail.python.org/pipermail/python-dev/2010-August/102915.html below:

[Python-Dev] Adding a token