#dev 2024-06-05

2024-06-05 UTC

[tantek] [snarfed] robots.txt is very coarse, and another invisible thing that many users may have no idea about, so I'd deliberately look for it and extra prompt confirm to the user that they wanted to override it if necessary