Repair longstanding damage introduced by the AST merge:
- Get rid of pgen.c and tokenizer_pgen.c. - Include tokenizer.c. All the -uall tests pass on my WinXP VC7.1 box now.
T
Tim Peters committed
903a9444a84b4af2c947e15d08968f9e990ae2bc
Parent: bb2bf2c