COMMITS
/ Lib/tokenize.py August 24, 2021
P
Add tests for the C tokenizer and expose it as a private module (GH-27924)
Pablo Galindo Salgado committed
July 31, 2021
P
bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499)
Pablo Galindo Salgado committed
January 24, 2021
A
bpo-43014: Improve performance of tokenize.tokenize by 20-30%
Anthony Sottile committed
May 30, 2019
A
bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)
Anthony Sottile committed
A
bpo-5028: fix doc bug for tokenize (GH-11683)
Andrew Carr committed
May 6, 2019
P
bpo-36766: Typos in docs and code comments (GH-13116)
penguindustin committed
December 22, 2018
S
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
Serhiy Storchaka committed
July 6, 2018
A
bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)
Ammar Askar committed
June 5, 2018
T
bpo-12486: Document tokenize.generate_tokens() as public API (#6957)
Thomas Kluyver committed
April 23, 2018
Ł
bpo-33338: [tokenize] Minor code cleanup (#6573)
Łukasz Langa committed
April 11, 2018
S
bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447)
Serhiy Storchaka committed
October 6, 2017
J
bpo-30406: Make async and await proper keywords (#1669)
Jelle Zijlstra committed
May 31, 2017
A
bpo-25324: copy tok_name before changing it (#1608)
Albert-Jan Nijburg committed
May 24, 2017
A
bpo-30377: Simplify handling of COMMENT and NL in tokenize.py (#1607)
Albert-Jan Nijburg committed
May 18, 2017
J
bpo-30296 Remove unnecessary tuples, lists, sets, and dicts (#1489)
Jon Dufresne committed
March 14, 2017
J
Add ELLIPSIS and RARROW. Add tests (#666)
Jim Fasarakis-Hilliard committed
September 9, 2016
B
Issue #26331: Implement the parsing part of PEP 515.
Brett Cannon committed
March 20, 2016
S
Issue #26581: Use the first coding cookie on a line, not the last one.
Serhiy Storchaka committed
S
Issue #26581: Use the first coding cookie on a line, not the last one.
Serhiy Storchaka committed
December 29, 2015
B
Issue #25977: Fix typos in Lib/tokenize.py
Berker Peksag committed
B
Issue #25977: Fix typos in Lib/tokenize.py
Berker Peksag committed
October 26, 2015
July 23, 2015
Y
Issue #24619: Simplify async/await tokenization.
Yury Selivanov committed
July 22, 2015
Y
Issue #24619: New approach for tokenizing async/await.
Yury Selivanov committed
June 28, 2015
J
Issue #20387: Merge test and patch from 3.4.4
Jason R. Coombs committed
June 22, 2015
D
Issue #20387: Restore retention of indentation during untokenize.
Dingyuan Wang committed
May 25, 2015
V
(Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary file
Victor Stinner committed
V
Issue #23840: tokenize.open() now closes the temporary binary file on error to
Victor Stinner committed
May 12, 2015
Y
PEP 0492 -- Coroutines with async and await syntax. Issue #24017.
Yury Selivanov committed
March 11, 2015
S
Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded with
Serhiy Storchaka committed
S
Issue #23615: Modules bz2, tarfile and tokenize now can be reloaded with
Serhiy Storchaka committed
January 11, 2015
S
Removed duplicated dict entries.
Serhiy Storchaka committed
December 5, 2014
V
Issue #22599: Enhance tokenize.open() to be able to call it during Python
Victor Stinner committed
V
(Merge 3.4) Issue #22599: Enhance tokenize.open() to be able to call it during
Victor Stinner committed
April 10, 2014
B
PEP 465: a dedicated infix operator for matrix multiplication (closes #21176)
Benjamin Peterson committed
February 24, 2014
T
Merge with 3.3
Terry Jan Reedy committed
T
whitespace
Terry Jan Reedy committed
T
Merge with 3.3
Terry Jan Reedy committed
T
Issue #9974: When untokenizing, use row info to insert backslash+newline.
Terry Jan Reedy committed
February 18, 2014
T
Merge with 3.3
Terry Jan Reedy committed
T
Issue #8478: Untokenizer.compat now processes first token from iterator input.
Terry Jan Reedy committed
February 17, 2014
T
Untokenize, bad assert: Merge with 3.3
Terry Jan Reedy committed
T
Untokenize: An logically incorrect assert tested user input validity.
Terry Jan Reedy committed
January 9, 2014
S
Issue #18960: Fix bugs with Python source code encoding in the second line.
Serhiy Storchaka committed
S
Issue #18960: Fix bugs with Python source code encoding in the second line.
Serhiy Storchaka committed
November 25, 2013
E
#19620: merge with 3.3.
Ezio Melotti committed
E
#19620: Fix typo in docstring (noticed by Christopher Welborn).
Ezio Melotti committed
September 16, 2013
S
Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
Serhiy Storchaka committed
S
Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
Serhiy Storchaka committed
December 25, 2012
A
Replace IOError with OSError (#16715)
Andrew Svetlov committed