The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.
copied from cf-staging / r-spiderbar
https://anaconda.org/conda-forge/r-spiderbar/badges/version.svg
https://anaconda.org/conda-forge/r-spiderbar/badges/latest_release_date.svg
https://anaconda.org/conda-forge/r-spiderbar/badges/latest_release_relative_date.svg
https://anaconda.org/conda-forge/r-spiderbar/badges/platforms.svg
https://anaconda.org/conda-forge/r-spiderbar/badges/license.svg
https://anaconda.org/conda-forge/r-spiderbar/badges/downloads.svg