2025-08-15 UTC
# 
[tantek]2 [social] it is not insanely hard. Google and other search engines solved the adaptive incremental crawl problem decade+ ago and I believe at this point the techniques are relatively well known. There is no excuse for the abusive rapid rescraping other than incompetent or lazy coding.