#dev 2022-11-28

2022-11-28 UTC
geoffo joined the channel
#
[snarfed] capjamesg looks like your site is duplicating the query params in the .well-known redirects
#
[snarfed] I definitely need to finish https://github.com/snarfed/bridgy-fed/issues/305
#
Loqi [snarfed] #305 Add debugging details to user page warnings?
jacky_ joined the channel
#
[ender] ideal: JavaScript `fetch` or `XHR` API call
#
[ender] is there a really really really easy way to display received webmentions for a self-hosted site? I read two blog posts about how to do this and i am still confused
#
[snarfed] [ender] https://webmention.herokuapp.com/ , https://indieweb.org/webmention.js , etc
#
Loqi A WebMention Endpoint
#
[ender] ++ snarfed
#
[ender] thanks!
#
[ender] oh no, didn't work...
#
[ender] snarfed++
#
Loqi snarfed has 44 karma in this channel over the last year (82 in all channels)
jacky_, gRegor, sp1ff and geoffo joined the channel
#
@jsonbecker ↩️ http://micro.blog, combines a static site generator with a Micropub API, full support for webmentions, RSS feed, and ALSO gives you the ability to follow (and generate) ActivityPub, you can do it from your own domain, and native crosspost to Tumblr/Twitter/Linkedin/Medium. (twitter.com/_/status/1597070947768733696)
jacky_, gRegorLove_, gRegorLove__, gRegor, geoffo, mro and kushal joined the channel
#
[capjamesg] ↩️ I didn't actually need to say Python here. I was going to say there are a lot of libraries that implement webmention functionality so if you ever need to do anything server-side with webmentions there's probably some code to help (I mentioned Python beacuse I worked on the Python webmention sender :D).
#
[capjamesg] ↩️ https://jamesg.blog/2022/10/21/send-webmention-in-python/
#
Loqi [James] Send a Webmention in 10 (or fewer) lines of Python code
#
capjamesg [ender] Side note: We try to avoid threads because they don't show up as well on other clients. Slack threads don't carry along easily to Discord, IRC, etc.
#
capjamesg [snarfed] What might cause that?
#
capjamesg I added the nginx directive in the docs. I did add it twice but I removed the second instance.
mro, gRegor and kloenk joined the channel
#
[KevinMarks] https://chaos.social/@selfawaresoup/109420916859423212 https://chaos.social/@selfawaresoup/109420936137227582
#
Loqi [Esther] This is amusing. Websites are struggling with the crawler requests from Fediverse instances when a new post propagates through the network. https://www.jwz.org/blog/2022/11/mastodon-stampede/Now that blogs and personal websites might see a revival, t...
#
[tantek] Or use simple PHP and no database
#
[tantek] What is thundering herd?
#
Loqi It looks like we don't have a page for "thundering herd" yet. Would you like to create it? (Or just say "thundering herd is ____", a sentence describing the term)
#
IWDiscordRelay <j​acky#7226> capjamesg: the capture from nginx might be duplicating it - had that issue before
#
IWDiscordRelay <c​apjamesg#4492> Capture?
jacky and [manton] joined the channel
#
[manton] @GWG Sorry I missed replying to your Micro.blog + feeds question the other day… I had jumped offline shortly after that. Anything that could be improved in Micro.blog?
#
aaronpk I don't *think* my site is falling over every time I send out an ActivityPub post but I should double check my caching situation
#
jacky I figure WebSub and Webmention would lighten this thundering herd effect, esp if they handle the caching from sites
#
jacky I guess you'd also hope that people would cache info re: webmention endpoints
#
sknebel how would websub or webmention "handle the caching"?
#
jacky if a websub hub sent pings with the page's content versus doing a thin ping (only just the content updated)
#
jacky I'd need to check the spec about this
#
jacky I think webmention senders could keep notes about who to send to for what (who: endpoints, what: site advertising endpoints)
#
sknebel websub sends fat pings, but this isnt about the post notification, but about context around the posts
#
sknebel ActivityPub is "fat pings" too after all
#
jacky hmm okay
#
jacky oh like reactions to things I've been informed about
#
jacky ugh
#
sknebel url previews in this case
#
sknebel instance gets a ping with a post that links to jwz -> instance fetches the linked page at a random point in the next minute
#
sknebel times a thousand instances -> some traffic
#
jacky hmm yikes
#
jacky is now lightly considering how he does previews in this case
#
jacky I can see the added benefit of the publisher adding the preview info inline (or somehow fetching a signed version of it from the URL to 'legitimatize' it somehow? eh), that sounds like the `Preview` activity
#
jacky yeah https://www.w3.org/TR/activitystreams-vocabulary/#dfn-preview
#
sknebel but I get why they dont want to trust random instances to deliver truthful previews
#
sknebel and isnt fetching a signed version from the url the same as fetching the url but with extra steps?
#
IWDiscordRelay <j​acky#7226> The signed version would be fetched from the publisher upon post creation and then maybe embedded into the post
#
IWDiscordRelay <j​acky#7226> I guess that still runs into the issue of the stampede
#
sknebel that sounds like what google tried with signed bundles for AMP 🙈
#
sknebel (also requires all web servers to support providing such signed versions...)
#
[schmarty] yeah this is an age-old problem of "here's the content i got from fetching this URL at this time i swear"
#
IWDiscordRelay <j​acky#7226> I have some relatively light caching on my site (keep for an hour and use etags based on checksums of some data)
#
IWDiscordRelay <j​acky#7226> Lol yeah
#
sknebel for something like this even microcaching (i.e. caches with expiry in single-digit seconds) helps
#
IWDiscordRelay <j​acky#7226> Hmm
#
aaronpk I just checked my config and I have something there called "microcache" but I can't tell how long the lifetime is 😅
#
aaronpk Maybe 10 seconds I think? Need to look up nginx docs again
geoffo joined the channel
#
[tantek] sknebel, precisely, and this is all way “fat pings” are kinda useless anyway, in practice everyone ends up fetching the original anyway because there’s always more use-cases for the original HTML than some minimized side plumbing
#
[tantek] What would/could help with thundering heard is content based addressing, e.g. something like AP
#
[tantek] Argh I mean IPFS
#
[KevinMarks] though cross-domain caching is not supported by browsers so you'd have to put it under a single domain anyway
#
[tantek] E.g. if a “fat ping” delivered both the https: and ipfs: URLs, and the receiver attempted ipfs lookup first, then they’d get a response from a local ipfs “cache” instead of burdening the original domain
#
[tantek] All the “thundering” would distribute IPFS caches of the content as a side effect
#
[tantek] [KevinMarks] it’s not the browsers that are doing the thundering, it’s servers fetching for link previews
#
[KevinMarks] Mastodon does something like that for the posts (each instance caches them and embedded media), it could do that for link previews for a url. Are we sure the thundering is not happening client side in mastodon?
chenghiz_ joined the channel
#
sknebel yes
#
[aciccarello] It's because each follower's instance makes a request
#
[KevinMarks] https://better.boston/@crschmidt/109416322650340296
#
Loqi [Christopher Schmidt] Additional fun things to learn: - Mastodon does not appear to use range headers or limit the size of a download to identify opengraph information, so linking a large file means 1000 servers are gonna read the whole damn thing? - Plemora does a HEAD...
#
[snarfed] heavily discussed in https://github.com/mastodon/mastodon/issues/4486#issuecomment-395076695, including how/whether to trust remote previews (conclusion: no), etc
#
Loqi [snarfed] i noticed this recently too. >1k requests in <45s, >25qps. not a disaster, my site handled it fine, but still, noticeable. small thread on it here: https://mastodon.technology/@snarfed/100119606571241751 , cc @ashfurrow @neekz0r. <img width="530" ...
jacky joined the channel
#
[tantek] 🍿
#
aaronpk wow that thread is a mess
#
aaronpk there's one good suggestion in there which I'm surprised got lost in the middle
#
aaronpk seems like fetching the link preview only when a person is looking at it makes the most sense
#
aaronpk sounds like right now the server fetches every url regardless of whether the post is actually seen by a person
#
jacky and that's def a waste of resources
#
sknebel that probably would distribute the flow from smaller instances at least
#
jacky but there's going to be someone who'll make the case that it'd be better to get it _before_ someone views a link versus after
#
jacky thinks waiting until they view a post is best but would need some sort of 'locking' I'd imagine
#
sknebel for a single user instance/reader starting to load it when its viewed probably means its even more likely the preview is never seen :D
#
sknebel unless you dynamically load it in as soon as its finished fetching
#
sknebel which has its own set of problems
#
aaronpk i forgot i actually customized my access log to include cache hit/miss/expired in it
#
jacky ooh that's a fast way to fill it :)
#
jacky unless it's like stats per line?
#
aaronpk no i just mean in addition to the log showing the URL and user agent and stuff, it also tells me whether the request was served from the cache
#
aaronpk useful to be able to tell if it's actually working
#
aaronpk just made a new URL that writes to a log file when the PHP code runs and confirmed that slamming that URL with requests only generates one log entry \o/
#
aaronpk thanks past me!
#
aaronpk https://media.aaronpk.com/2022/11/28080947-5576.png
jacky joined the channel
#
aaronpk time for a blog post i guess
#
[tantek] yup
#
sknebel What is microcaching?
#
Loqi It looks like we don't have a page for "microcaching" yet. Would you like to create it? (Or just say "microcaching is ____", a sentence describing the term)
#
aaronpk microcaching is a technique of caching a web server response for a very short period of time to reduce the effects of the [[thundering herd]] problem
#
Loqi ok
#
[tantek] good thing it's not about caching microformats or microservices 🤡
#
[snarfed] Yeah granary Bridgy etc do that a lot
geoffo joined the channel
#
[ender] capjamesg++
#
Loqi capjamesg has 31 karma in this channel over the last year (89 in all channels)
gRegor, jacky and [jacky] joined the channel
#
[KevinMarks] more tips on scaling Mastodon https://hazelweakly.me/blog/scaling-mastodon/
#
jacky oh they're good people
#
jacky would love to lure them
#
jacky wow yeah the nginx cheat to object store is nifty
mro joined the channel
#
GWG [manton]: I'm not sure. How would you want a system of posse to Micro.blog via Micropub syndication targets to work?
neceve joined the channel
#
[manton] @GWG I’m not sure either… I need to review how folks are doing things on their sites right now.
#
GWG [manton]: Worth experimenting? I'm working on moving the Bridgy code to use Micropub over Webmention posse which is why I'm looking at everything
neceve joined the channel
#
IWDiscordRelay <j​acky#7226> I have yet to move to that
#
IWDiscordRelay <j​acky#7226> my hope is to use that to make syndication a bit more 'invisible', i.e: I could sign into something like bridgy using my site, check some boxes on the kind of places I'd like to publish to and go from there
#
IWDiscordRelay <j​acky#7226> it _is_ a bit willy-nilly but I'd added support to /Koype to allow for adding syndication targets over Micropub using a custom action `add-syndication-target` (in the spirit of https://github.com/indieweb/micropub-extensions/issues/40#issuecomment-1011492547, still working on it at https://git.jacky.wtf/indieweb/koype-next/issues/4)
#
Loqi [jalcine] Yeah! So my hope/goal is to allow for the creation of channels and also allow for querying for more information about them (a _bit_ similar to how we do categories right now). The two cases I gave above are very specific to the content rendering c...
#
jacky I'm hoping to propose it as an idea once I get a client going that isn't Koype itself to add such targets
#
[KevinMarks] Instance probing https://infosec.exchange/@malwaretech/109423683872990336
#
Loqi [Marcus Hutchins :verified:] I wonder if this will work https://test.malwaretech.com/mastodon.php
#
[KevinMarks] (only works when viewed in mastodon timeline)
#
aaronpk https://media.aaronpk.com/2022/11/28142203-5801.png nice
#
aaronpk `curl -H "User-Agent: http://hacked.example/" https://test.malwaretech.com/mastodon.php`
#
aaronpk hah cloudflare didn't like the <script> I included in the user agent
neceve joined the channel