protego
A pure-Python robots.txt parser with support for modern conventions
- Overview
- Files 8
- Labels 1
- Badges
Installation
To install this package, run one of the following:
Usage Tracking
0.4.0
1 / 8 versions selected
Total downloads: 0
Description
A pure-Python robots.txt parser with support for modern conventions like the one defined by Google.
About
Summary
A pure-Python robots.txt parser with support for modern conventions
Information Last Updated
Sep 8, 2025 at 17:28
License
BSD-3-Clause
Total Downloads
8
GitHub Repository
https://github.com/scrapy/protegoDocumentation
https://github.com/scrapy/protego