#dev 2025-08-15
2025-08-15 UTC
duanin2, grufwub, CRISPR and geoffo joined the channel
# 
paperbritt Sure! Krystal (my hosting) have recommended settings that I mostly just followed but I am starting to understand what they do. Perhaps if someone else sees the page once done they can add more knowledge to that.
oodani, cotsuka, Xe_ and doesnm joined the channel
# 
shminge I've gone from not even using RSS a week ago to implementing my own from scratch on my site today
[schmarty] and doesnm joined the channel
SDavidPrince and benji- joined the channel
[morganm], balintm and [social] joined the channel
# [snarfed] ah these are all older posts, looks like you're getting bit by https://indieweb.social/@jamesg.blog@jamesg.blog
# 
capjamesg Like https://jamesg.blog/syndicated/2025/01/01/example that redirects to the same URL without /syndicated/
luca joined the channel
balintm and duanin2 joined the channel
luca joined the channel
ben, gRegor and [aciccarello] joined the channel
# 
[aciccarello] I've only used it for my RSS feed https://www.ciccarello.me/blog/2021/04/17/styling-an-rssatom-feed-with-xsl/
# 
[artlung] Finally used the `template` element. Using it to define the contents of a youtube embed popup. https://artlung.com/blogroll/ (and I handle arrow navigation which I don't currently expose)
# 
balintm i'm using it for my feeds as well https://bm.gy/feed
# balintm oh and my sitemap :) https://bm.gy/sitemap
ben__, duanin2 and barnaby joined the channel
cotsuka, ttybitnik, balintm, Saige_Leah[d], mahboubine and barnaby joined the channel
# mathium05 lol
# [tantek]2 what worries me about this long term is whether "LLM overcrawling" is pushes the web to get behind a very small number of CDNs/hosts, the same way that "overwhelming spam" pushed email to get behind a small number of providers.
# [tantek]2 is what* pushes the web
# [aciccarello] I think some organizations should just use a shared dataset like https://commoncrawl.org/. But I do worry about the issue of scale leading to less competition like we have in the search engine space.
streety and lain`_ joined the channel
# [social] I keep thinking there should be one central resource for LLM crawling. As site owners you allow access, deny access, or allow access to crawl with payment and approved contract for use.
# [social] This is just wild west and everybody in town is a cattle rustler.
# [social] [aciccarello]++
cotsuka joined the channel
# [tantek]2 [social] I feel like we're way past anyone taking "one central resource" seriously for anything to do with the internet/web. I mean that's fundamentally against foundational principles of both. Nevermind that no one trusts anyone else to be the "one central resource"
# lain`_ the icann?
# [tantek]2 The closest we have to such "central resources" are standards bodies, and even with those we try to limit their scope and powers
# [tantek]2 and still they are all subject to criticism and varying levels of "trust". yes, icann included
# lain`_ i wonder if some kind of automated system that can detect malicious ips and submit them to a blacklist after enough 'votes' or something would work
luca joined the channel
# [tantek]2 then who adjudicates the blocklist?
# [tantek]2 or handles appeals for mistakes?
# lain`_ hmm
# lain`_ simply make it good enough that there's not any mistakes
# [tantek]2 yeah it's a hard problem
# [tantek]2 that is quite odd of a tell
duanin2 and gRegor joined the channel
# [social] [tantek] I agree with that the too late for central resource. At the same time most everything has been scraped a few years ago, and yet it all continues.
# [social] It is an insanely hard problem and sticky one.
# 
gRegor I've been getting regular form-spam submissions on my passwordless sign-in page. Trying to avoid captchas and the like, so I've started tracking IPs and denying them after a set number of failures. I have CSRF and a honeypot field on it, which catch some, but seem to catch a lot less than in the past.
# [tantek]2 [social] it is not insanely hard. Google and other search engines solved the adaptive incremental crawl problem decade+ ago and I believe at this point the techniques are relatively well known. There is no excuse for the abusive rapid rescraping other than incompetent or lazy coding.
# [tantek]2 Perhaps they vibe-coded their crawler 🙃
# [tantek]2 crawlers*
# [social] Technically not hard, but there is no body with oversight. I agree with that, and that isn’t what we are talking about. It is the mis-use and there lacking a body to have push back.
# [tantek]2 yeah both that (lack of oversight) and, frankly, lack of proper incentives designed into the protocols to organically discourage this kind of misbehavior
# [tantek]2 this is why I made the comparison to SMTP
# [social] Having done work monitoring and prepping the US delegation for ITU and other bodies (as well as US entities) that sort of oversight and ability to push back (to some degree) isn’t there. Having that sort of oversight around actors on the internet (ITU and others isn’t strong, but it helps), or the lack of it has always seemed odd.
# [social] Without oversight it is whack-a-mole. That is where it gets difficult. Big players can take steps and often the right steps, to control their overstepping with crawling. But, it is the other players.
# [tantek]2 the flipside of that is the when governments do the oversight, they seem to get it wrong far too often. e.g. SOPA/PIPA and most recently the UK anti-encryption thing and age thing
# [social] Agreed