2021-10-06 UTC
# 
capjamesg[d] Thanks for the content filtering discussion everyone! I made one change: sites marked as "adult" will not be indexed, according to a meta tag. Google uses this in their SafeSearch to the extent I am aware. Code for reference: https://github.com/capjamesg/indieweb-search/blob/main/crawler/url_handling.py#L252