COMMITS
/ Doc/library/tokenize.rst February 19, 2026
N
Docs: remove unneeded author attributions (#145002)
Ned Batchelder committed
February 6, 2026
N
Docs: module pages should not link to themselves (#144505)
Ned Batchelder committed
January 20, 2025
T
May 8, 2024
N
docs: module page titles should not start with a link to themselves (#117099)
Ned Batchelder committed
October 3, 2023
A
Docs: Avoid the deprecated ``.. cmdoption::`` directive (#110292)
Adam Turner committed
June 9, 2023
P
Clarify the supported cases in the tokenize module (#105569)
Pablo Galindo Salgado committed
June 7, 2023
P
gh-105390: Correctly raise TokenError instead of SyntaxError for tokenize errors (#105399)
Pablo Galindo Salgado committed
March 30, 2020
M
fix comma location in various places (GH-19233)
Mathieu Dupuy committed
January 25, 2020
W
bpo-36654: Add examples for using tokenize module programmically (#12947)
Windson yang committed
May 30, 2019
A
bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)
Anthony Sottile committed
A
bpo-5028: fix doc bug for tokenize (GH-11683)
Andrew Carr committed
June 5, 2018
T
bpo-12486: Document tokenize.generate_tokens() as public API (#6957)
Thomas Kluyver committed
April 8, 2018
S
Improve highlighting of some code blocks. (GH-6401)
Serhiy Storchaka committed
September 13, 2017
M
bpo-31394: Make tokenize.rst PEP 8-compliant (GH-3526)
Mariatta committed
A
bpo-31394: Clarify documentation about token type attribute (GH-3469)
Aivar Annamaa committed
June 6, 2017
S
bpo-25324: Move the description of tokenize tokens to token.rst. (#1911)
Serhiy Storchaka committed
June 11, 2016
T
Issue #22558: Add remaining doc links to source code for Python-coded modules.
Terry Jan Reedy committed
May 10, 2016
S
Issue #23921: Standardized documentation whitespace formatting.
Serhiy Storchaka committed
January 16, 2016
M
Issue #26127: Fix links in tokenize documentation; patch by Silent Ghost
Martin Panter committed
November 2, 2015
S
Issue #25523: Further a-to-an corrections.
Serhiy Storchaka committed
June 8, 2014
B
document TokenError and unclosed expression behavior (closes #12063)
Benjamin Peterson committed
January 19, 2012
M
Issue #2134: Add support for tokenize.TokenInfo.exact_type.
Meador Inge committed
October 7, 2011
M
Issue #12943: python -m tokenize support has been added to tokenize.
Meador Inge committed
January 10, 2011
R
Separate source link from main text.
Raymond Hettinger committed
R
Move source links to consistent location and remove wordy, big yellow boxes.
Raymond Hettinger committed
November 16, 2010
É
Provide links to Python source where the code is short, readable and
Éric Araujo committed
November 9, 2010
V
Issue #10335: Add tokenize.open(), detect the file encoding using
Victor Stinner committed
September 9, 2010
R
Improve the repr for the TokenInfo named tuple.
Raymond Hettinger committed
March 18, 2010
B
show a common usage of detect_encoding
Benjamin Peterson committed
B
in tokenize.detect_encoding(), return utf-8-sig when a BOM is found
Benjamin Peterson committed
January 4, 2010
A
Issue #7631: Fix undefined references to the "built-in file object", which
Antoine Pitrou committed
December 20, 2009
E
Fixed capitalization and markup; added imports in the example
Ezio Melotti committed
April 29, 2009
R
Issue #5857: tokenize.tokenize() now returns named tuples.
Raymond Hettinger committed
March 13, 2009
G
#5486: typos.
Georg Brandl committed
January 3, 2009
G
Remove trailing whitespace.
Georg Brandl committed
June 10, 2008
March 18, 2008
T
- Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been
Trent Nelson committed
February 22, 2008
November 1, 2007
G
#1370: Finish the merge r58749, log below, by resolving all conflicts in Doc/.
Georg Brandl committed
September 4, 2007
G
Convert all print statements in the docs.
Georg Brandl committed
September 1, 2007
G
Get rid of the remaining versionadded/versionchanged directives.
Georg Brandl committed
August 15, 2007
G
Move the 3k reST doc tree in place.
Georg Brandl committed