COMMITS
/ Lib/urllib/robotparser.py September 5, 2025
S
gh-88375, gh-111788: Fix parsing errors and normalization in robotparser (GH-138502)
Serhiy Storchaka committed
January 13, 2025
B
Explicitly import `urllib.error` in `urllib.robotparser` (#128737)
Bénédikt Tran committed
January 12, 2025
S
gh-128731: Fix ResourceWarning in robotparser.RobotFileParser.read() (GH-128733)
Serhiy Storchaka committed
June 16, 2019
May 16, 2018
C
bpo-21475: Support the Sitemap extension in robotparser (GH-6883)
Christopher Beacham committed
May 14, 2018
M
bpo-32861: urllib.robotparser fix incomplete __str__ methods. (GH-5711)
Michael Lazar committed
November 23, 2017
B
bpo-31325: Fix usage of namedtuple in RobotFileParser.parse() (#4529)
Berker Peksag committed
September 18, 2016
B
May 8, 2016
M
Merge typo fixes from 3.5
Martin Panter committed
M
Fix typos in comments, documentation and test method names
Martin Panter committed
October 8, 2015
B
Issue #16099: RobotFileParser now supports Crawl-delay and Request-rate
Berker Peksag committed
May 13, 2014
R
Issue 21469: Minor code modernization (convert and/or expression to an if/else expression).
Raymond Hettinger committed
R
Issue 21469: Mitigate risk of false positives with robotparser.
Raymond Hettinger committed
May 29, 2013
S
#17403: urllib.parse.robotparser normalizes the urls before adding to ruleline.
Senthil Kumaran committed
July 29, 2010
G
#4108: the first default entry (User-agent: *) wins.
Georg Brandl committed
July 28, 2010
S
Fix Issue6325 - robotparse to honor urls with query strings.
Senthil Kumaran committed
July 31, 2008
B
Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from
Benjamin Peterson committed
July 18, 2008
J
Bug 3347: robotparser failed because it didn't convert bytes to string.
Jeremy Hylton committed
June 18, 2008
J
Make a new urllib package .
Jeremy Hylton committed