2019-07-12 UTC
# 
sknebel [snarfed]: e.g. https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html "Disallow in robots.txt: Search engines can only index pages that they know about, so blocking the page from being crawled usually means its content won’t be indexed. While the search engine may also index a URL based on links from other pages, without seeing the content itself, we aim to make such pages less visible in the