#dev 2021-09-01

2021-09-01 UTC

[fluffy] per the robots spec, all 4xx errors on retrieving robots.txt should be treated as 404 (i.e. “there’s no robots.txt”), and it’s a 5xx that should be treated as “you are being disallowed globally”