COMMITS
/ Lib/tokenize.py December 8, 2025
V
Fix pyflakes warnings: variable is assigned to but never used (#142294)
Victor Stinner committed
October 20, 2025
S
gh-63161: Fix tokenize.detect_encoding() (GH-139446)
Serhiy Storchaka committed
May 26, 2025
May 25, 2025
L
May 5, 2025
H
gh-130645: Add color to stdlib argparse CLIs (gh-133380)
Hugo van Kemenade committed
May 3, 2025
S
gh-133306: Use \z instead of \Z in regular expressions in the stdlib (GH-133337)
Serhiy Storchaka committed
April 30, 2025
L
gh-132661: Implement PEP 750 (#132662)
Lysandros Nikolaou committed
March 27, 2025
S
gh-131178: Add tests for `tokenize` command-line interface (#131274)
Semyon Moroz committed
January 21, 2025
T
gh-125553: Fix backslash continuation in `untokenize` (#126010)
Tomas R. committed
January 6, 2025
T
gh-128519: Align the docstring of untokenize() to match the docs (#128521)
Tomas R. committed
October 9, 2024
S
gh-91818: Use default program name in the CLI of many modules (GH-124867)
Serhiy Storchaka committed
October 6, 2024
T
gh-125008: Fix `tokenize.untokenize` roundtrip for `\n{{` (#125013)
Tomas R. committed
February 19, 2024
P
gh-115154: Fix untokenize handling of unicode named literals (#115171)
Pablo Galindo Salgado committed
October 12, 2023
L
gh-104169: Fix test_peg_generator after tokenizer refactoring (#110727)
Lysandros Nikolaou committed
June 19, 2023
L
gh-71299: Fix __all__ in tokenize (#105907)
Lysandros Nikolaou committed
June 7, 2023
P
gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors (#105399)
Pablo Galindo Salgado committed
June 5, 2023
P
gh-105324: Fix tokenize module main function for stdin (#105325)
Pablo Galindo Salgado committed
May 30, 2023
P
gh-105069: Add a readline-like callable to the tokenizer to consume input iteratively (#105070)
Pablo Galindo Salgado committed
May 26, 2023
P
gh-104976: Ensure trailing dedent tokens are emitted as the previous tokenizer (#104980)
Pablo Galindo Salgado committed
May 22, 2023
M
gh-102856: Tokenize performance improvement (#104731)
Marta Gómez Macías committed
May 21, 2023
M
gh-104719: Restore Tokenize module constants (#104722)
Marta Gómez Macías committed
M
gh-102856: Python tokenizer implementation for PEP 701 (#104323)
Marta Gómez Macías committed
February 3, 2022
N
bpo-46565: `del` loop vars that are leaking into module namespaces (GH-30993)
Nikita Sobolev committed
August 24, 2021
P
Add tests for the C tokenizer and expose it as a private module (GH-27924)
Pablo Galindo Salgado committed
July 31, 2021
P
bpo-44667: Treat correctly lines ending with comments and no newlines in the Python tokenizer (GH-27499)
Pablo Galindo Salgado committed
January 24, 2021
A
bpo-43014: Improve performance of tokenize.tokenize by 20-30%
Anthony Sottile committed
May 30, 2019
A
bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)
Anthony Sottile committed
A
bpo-5028: fix doc bug for tokenize (GH-11683)
Andrew Carr committed
May 6, 2019
P
bpo-36766: Typos in docs and code comments (GH-13116)
penguindustin committed
December 22, 2018
S
bpo-30455: Generate all token related code and docs from Grammar/Tokens. (GH-10370)
Serhiy Storchaka committed
July 6, 2018
A
bpo-33899: Make tokenize module mirror end-of-file is end-of-line behavior (GH-7891)
Ammar Askar committed
June 5, 2018
T
bpo-12486: Document tokenize.generate_tokens() as public API (#6957)
Thomas Kluyver committed
April 23, 2018
Ł
bpo-33338: [tokenize] Minor code cleanup (#6573)
Łukasz Langa committed
April 11, 2018
S
bpo-33260: Regenerate token.py after removing ASYNC and AWAIT. (GH-6447)
Serhiy Storchaka committed
October 6, 2017
J
bpo-30406: Make async and await proper keywords (#1669)
Jelle Zijlstra committed
May 31, 2017
A
bpo-25324: copy tok_name before changing it (#1608)
Albert-Jan Nijburg committed
May 24, 2017
A
bpo-30377: Simplify handling of COMMENT and NL in tokenize.py (#1607)
Albert-Jan Nijburg committed
May 18, 2017
J
bpo-30296 Remove unnecessary tuples, lists, sets, and dicts (#1489)
Jon Dufresne committed
March 14, 2017
J
Add ELLIPSIS and RARROW. Add tests (#666)
Jim Fasarakis-Hilliard committed
September 9, 2016
B
Issue #26331: Implement the parsing part of PEP 515.
Brett Cannon committed
March 20, 2016
S
Issue #26581: Use the first coding cookie on a line, not the last one.
Serhiy Storchaka committed
S
Issue #26581: Use the first coding cookie on a line, not the last one.
Serhiy Storchaka committed
December 29, 2015
B
Issue #25977: Fix typos in Lib/tokenize.py
Berker Peksag committed
B
Issue #25977: Fix typos in Lib/tokenize.py
Berker Peksag committed
October 26, 2015
July 23, 2015
Y
Issue #24619: Simplify async/await tokenization.
Yury Selivanov committed
July 22, 2015
Y
Issue #24619: New approach for tokenizing async/await.
Yury Selivanov committed
June 28, 2015
J
Issue #20387: Merge test and patch from 3.4.4
Jason R. Coombs committed
June 22, 2015
D
Issue #20387: Restore retention of indentation during untokenize.
Dingyuan Wang committed
May 25, 2015
V
(Merge 3.5) Issue #23840: tokenize.open() now closes the temporary binary file
Victor Stinner committed