2022-09-16 UTC
# 
[tonz] [snarfed] wrt ‘it may add up to significant bandwidth for big sites’ , an example given was a single page wordpress site w 4 files, getting single digit visitors per week, content not changing, still getting 600K hits from bots and crawlers per month. In part because of all those unnecessary URLs (and crawlers being wasteful themselves) A big site saw Google crawler hit their site every 2 seconds to index the whole thing. So it’