#dev 2021-09-01

2021-09-01 UTC

[fluffy] and our response is always the same, “we need to be able to see the robots.txt to be able to process it, and the robots spec says to return a 500 error if you want us to treat it as a global deny”