#dev 2021-07-16

2021-07-16 UTC

capjamesg Basic logic is this: find URLs in sitemap, crawl page, if new URL is discovered I want to add it to the "to crawl" list, add information about page to DB, go on to next. And so on and so on.