protego
A pure-Python robots.txt parser with support for modern conventions
A pure-Python robots.txt parser with support for modern conventions
To install this package, run one of the following:
A pure-Python robots.txt parser with support for modern conventions like the one defined by Google.
Summary
A pure-Python robots.txt parser with support for modern conventions
Information Last Updated
Sep 8, 2025 at 17:28
License
BSD-3-Clause
Total Downloads
8
GitHub Repository
https://github.com/scrapy/protegoDocumentation
https://github.com/scrapy/protego