r-robotstxt
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Installation
To install this package, run one of the following:
Usage Tracking
0.7.13
1 / 8 versions selected
Downloads (Last 6 months): 0
About
Summary
Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.
Last Updated
Jan 16, 2024 at 22:29
License
MIT
Total Downloads
36
Version Downloads
36
Supported Platforms
noarch