2021-09-01 UTC
# 
sknebel (e.g. google says about 5xx " Because the server couldn't give a definite response to Google's robots.txt request, Google temporarily interprets server errors as if the site is fully disallowed [...] If the robots.txt is unreachable for more than 30 days, Google will use the last cached copy of the robots.txt. If unavailable, Google assumes that there are no crawl restrictions. ")