#SaphireI'll need to chew on the dns records later
[sebsel] joined the channel
#sknebelhah, found some ticketing code in my site that I didnt remember adding. apparently I sent some tickets to GWG at some point :D
#sgregerAre there any opinions/precedence re respecting robots.txt when polling several thousand URLs (from my own content) to check for their existence?
[bjoern] joined the channel
#sgregerI'm testing a link checker tool to validate outgoing links on my old content and getting a huge amount of "Access denied by robots.txt". Feels right to obey, but also renders this task almost pointless.
#Saphiresgreger: how much of it is thanks to ad farms?
#sknebelsgreger: I feel like a linkchecker you individually run should not respect robots.txt
#SaphireRobots.txt is more for automated scrapping and aggregation
#sgregerSaphire: I concur. Presumably, most robots.txt excludes are in place to discourage other actors than genuine link verification.
#SaphireIt might be useful as a signal "that might not be a webpage you are looking for" though
#sgregersknebel: It's a bit of a conundrum. Technically it's a bot, but what it does is akin of me going through old links for both side's benefit.
#sgregerThis entire automated validation of link-rot is a bit of a rabbit hole, I just realize. A lot of "Schrodinger's URLs": timeouts, rate limits, potentially temporary certificate issues, HTTP200s but with totally different owners/content, etc.
#GWGZegnat: Can I pick your brain on a question that we discussed in 2021-ish?
#GWGZegnat: Specifically this one: https://github.com/indieweb/indieauth/issues/127 But there is a broader IndieAuth question...should the issuer URL have the metadata header on it...so you could find the metadata endpoint when looking at the issuer URL? Mine doesn't.
#Loqi[preview] [dshanske] #127 Where does the ticket endpoint go looking for the token endpoint?
IWSlackGateway and [tantek] joined the channel
#ZegnatI think I will need some time to read up, and I am currently with my head in a different subject at create day, so not sure I can actually give you anything smart right now
[KevinMarks] joined the channel
#[KevinMarks]"Does the response still contain the text I quoted? " would maybe be a better test