A generic concurrent tree crawling algorithm with a sample implementation for website crawling.