bpo-12486: Document tokenize.generate_tokens() as public API (#6957)
* Document tokenize.generate_tokens() * Add news file * Add test for generate_tokens * Document behaviour around ENCODING token * Add generate_tokens to __all__
T
Thomas Kluyver committed
c56b17bd8c7a3fd03859822246633d2c9586f8bd
Parent: c2745d2
Committed by Carol Willing <carolcode@willingconsulting.com>
on 6/5/2018, 5:26:39 PM