2021-08-30 UTC
# 
[tantek] crawling and searching has always seemed like a natural fit for distributed computing, so I have to admit I'm always perplexed when the default is to build one thing that crawls everything (instead of coming up with a protocol to do distributed (perhaps self-?) crawling and maybe even results retrieval, leaving aggregation/ranking up to specific UIs