r-robotstxt
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
- Overview
- Files 1
- Labels 1
- Badges
Installation
To install this package, run one of the following:
Usage Tracking
0.7.13
1 / 8 versions selected
Total downloads: 0
About
Summary
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Information Last Updated
Apr 22, 2025 at 15:31
License
MIT
Total Downloads
34
Platforms
noarch Version: 0.7.13