[snarfed]I find it ironic, if not a bit disingenous, that the fediverse fights so hard against search engines and other centralized indexing, and yet when I spot check of a few of the biggest Mastodon instances' robots.txt files, they don't even bother to block crawlers there
