#dev 2025-08-14
2025-08-14 UTC
grufwub, srxl, jeremy and rrix joined the channel
# 
[social] gRegor I sorted my robots.txt issue as I had two I saved in BBEdit with one for my dev. sub-domain and the other from my main site. I didn’t follow-up on my note to self to add the “s” back on the one for my main site.
NaomiAmethyst, [morganm], oodani, vidak, [KevinMarks], jjuran, doesnm, [Murray]1, [tantek]2, [artlung]1, [mattl]2, [aciccarello]1, [Jo]1, [KevinMarks]1, [social]1, [morganm]1, [Jim_Mitchell]1, [snarfed]1, IWSlackGateway4, [schmarty]2 and [Ana_R] joined the channel
oodani joined the channel
# 
[KevinMarks] I get those, and I sometimes reply "what did you search?" given I have ranked first for my name for over 20 years and once started a project at Google to help people find other people named Kevin Marks
rossabaker, [lazcorp], duanin2, ttybitnik, brentlineberry, lanodan, NaomiAmethyst8 and NaomiAmethyst joined the channel
# IWDiscord <paperbritt>
# 
paperbritt Moving this to dev at the request of our friendly and respectful bot. Any suggestions are very welcome.
rozodru joined the channel
# 
capjamesg [edit] It may be https://developers.cloudflare.com/bots/get-started/bot-fight-mode/
# [tantek]2 I feel like this is something more and more web hosts are going to have to figure out on behalf of their customers because it's too much to expect every site admin to do this manually (or not great to have everyone "just use Cloudflare")
# 
aaronpk like imagine instead of just yes/no, it's "block all AI crawlers except chatgpt.com" https://media.aaronpk.com/2025/08/14065535-3779.png
# aaronpk oh wow this is way farther along than I thought it was https://developers.cloudflare.com/bots/concepts/bot/verified-bots/web-bot-auth/
# paperbritt I've avoided setting up a CDN because it's just another thing to maintain, especially for a site usually with daily traffic well under 100.
# paperbritt That's good to know.
# paperbritt Is it possible the same AI scrapers have come via indieweb.org? Using the wiki user list to find their next websites? If so it's probably worth everyone else keeping an eye on their traffic.
doesnm joined the channel
# paperbritt The timing with the problems there is odd.
# paperbritt My site isn't worth the money running these bots is probably costing someone.
# paperbritt Wow.
[aaronpk] and ben__ joined the channel
# 
[artlung] paperbritt are there caching layers available to you for your site? I'm seeing this page in the woocommerce docs https://developer.woocommerce.com/docs/best-practices/performance/configuring-caching-plugins/
ttybitnik joined the channel
# paperbritt I've got litespeed cache set up, fully supported by the host with redis for the object cache. The litespeed cache plugin has a few settings specifically for WooCommerce but I honestly struggle with the amount of settings between that and wordfence, so if you told me something was misconfigured I wouldn't be surprised.
gRegor, barnaby, sp1ff, jonnybarnes, lanodan, doesnm, gregsarjeant, eb, vidak, sebsel, geoffo_, perryflynn, jjuran, voxpelli, rhiaro, pfych, ramsey, okCiel, opencircuit, GWG, Virtual, Tiffany, dustinm`, oodani, Pixi, claudinec, srushe, nnrx, _123nick[d], Saige_Leah, athenaeryma, Lars-Christian, aaronpk, ben, ancarda, dissolve22[d], vikanezrimaya, streety, Saige_Leah[d] and CRISPR joined the channel
# [tantek]2 what is litespeed
# 
Loqi LiteSpeed (AKA LightSpeed) [HTTPD] is Apache drop-in replacement commercial web server software https://indieweb.org/Litespeed
# [tantek]2 paperbritt, if you have a link/reference to docs for litespeed cache, perhaps you could add a brief citation / "How to setup cache" subsection on that page ^ ? Thanks!