2021-09-15 UTC
# 
capjamesg[d] I could read the robots.txt / sitemap.xml resources for a couple of sites at once and then use concurrent.futures thread pool to create threads for each site. That way I could have five threads working on five sites.