A pure-Python robots.txt parser with support for modern conventions
conda install anaconda::protego
A pure-Python robots.txt parser with support for modern conventions like the one defined by Google.