A generic concurrent tree crawling algorithm with a sample implementation for website crawling.
anaconda login
conda install auto::concurrent_tree_crawler