#dev 2023-06-23
2023-06-23 UTC
saptaks and oxtyped joined the channel
# [snarfed] interesting, Mastodon now shows verified rel-me links in search results: https://snarfed.org/mastodon-search.png
neatnik joined the channel
# [snarfed] I searched https://socialhub.activitypub.rocks/ and https://github.com/mastodon/mastodon/issues just now, didn't find anything obvious
# [snarfed] closest was https://github.com/mastodon/mastodon/issues/20159#issuecomment-1307861787 , which says that each instance does its own verification of remote users' profile links. which still begs the question of how we trigger that verification from non-Mastodon AP
[jacky] joined the channel
btrem and willnorris_ joined the channel
# vladimyr It is very simple service and only thing you really need to know is that Nokogiri is DOM implementation in ruby enabling you to query selectors and xpaths on provided html string
# vladimyr This gets triggered each time you update your mastodon profile and is completely independent from activitypub side of things
# vladimyr This is mastodon specific
strugee joined the channel
# vladimyr I mean when you go and add or change links at your mastodon profile which is not something you can do by activitypub message/call
# vladimyr Oh you were talking about remote profile verification, well haven't looked into the code but I'd assume it just calles same sidekiq service with account info exchanged with remote instance
# vladimyr It probably gets triggered each time you do the search with some cache involved cause I know for a fact that your own profile links are also periodically re-verified simply due to volatile nature of urls
tei_ joined the channel
# vladimyr That's relatively easy to check, just host somewhere fake actpub actor
# vladimyr Only json blob describing the user
# vladimyr Search for that url
# vladimyr And put inside urls you want to verify
# vladimyr He tried sending update activity
# vladimyr If I understood correctly
# vladimyr What I'm saying is do the foo.example.com/actor.json and put inside that json reference to bar.example.com
# vladimyr Host index.html on bar.example.com pointing back to foo.example.com/actor.json
# vladimyr And search on your masto instance for foo.example.com/actor.json
# vladimyr That should be enough to force mastodon to take that actpub blob, extract profile link and start verificafion dance?
# vladimyr Also this "validation" logic is blissfully naive and not to be really trusted
# vladimyr Cause it works by
# vladimyr 1. find all rel=me links and anchors
# vladimyr 2. If it matches canonical actor url it is verified
# vladimyr 3. If not take only first of them, do the head request and check if it redirects to cannonical actor url
# [snarfed] right, I understand what you're proposing. and aaronpk didn't describe what he tried: https://chat.indieweb.org/dev/2023-06-23#t1687483479551500
# vladimyr (not sure why only the first one is checked)
# vladimyr Because bad people can be creative in such manner that they serve different responses for head and get requests
strugee joined the channel
# vladimyr Um, let's see, I need two domains/locations, one responding with actpub actor json and another with html and instance from which I need to trigger search
# vladimyr I can try on any public instance
# vladimyr And for quick and diry test I could use replit.com or val.town or anything like that
# vladimyr Shouldn't require too much code
# vladimyr And I have example from screenshot to guide me
# vladimyr (also you are right, it is on user if they mess up verification I'm just saying I'm not a fan of switching http method cause it makes it harder to debug)
# [snarfed] I set it up just now too. my Bridgy Fed account @snarfed.org@snarfed.org has a profile link to https://snarfed.org/, which now has rel-me links back to both the AS2 actor id https://fed.brid.gy/snarfed.org and its url https://fed.brid.gy/r/https://snarfed.org/ . we'll see if it gets a green check anywhere
# vladimyr (and it is not entirely up to /standard/ cause it misses http link headers which should be interchangeable with link elements)
strugee_ joined the channel
# vladimyr Did I miss something?
# vladimyr Lol
# vladimyr We all miss something Loqi :D
# vladimyr [snarfed]: great! (Side note: matrix bridge is flaky so if sometimes seems I ignore what you are saying that's because I literally didn't get those messages :/)
# vladimyr Exactly but I'll manage by switching between matrix and chat archive
alecjonathon_, angelo, eitilt, jan6, jeremycherfas, omz13, sknebel, pharalia, chenghiz_, vikanezrimaya, ancarda, darylsun, eb_, alecjonathon, capjamesg, vikanezrimaya_, srushe, eb, Xe, strugee_, Ruxton, benji, superkuh, jjuran, ben_thatmustbeme and moose333 joined the channel
# vladimyr Um [snarfed] I'm doing some debugging and according to my very scientific tests your account is /webfingerable/ only from acct: uri?
aaronpk, Seirdy, jonnybarnes, ludovicchabant, Kaja and gnoo joined the channel
# vladimyr Scratch that, I can webfinger snarfed.org but it serves redirect
gnoo, superkuh, ludovicchabant, Kaja, ben_thatmustbeme, jjuran, Seirdy, jonnybarnes, aaronpk, moose333, benji, sknebel, chenghiz_, omz13, jeremycherfas, eitilt, Ruxton, angelo, IWDiscord, Saphire, laker, sebsel, vilhalmer, Seirdy_, [0x3b0b], vladimyr, bkil, olaf[m], klez, voxpelli, mretka, joshproehl, sebbu, epoch_, gwg_, sivoais, Zegnat, trwnh, petermolnar, IWSlackGateway, [Ana_R], [schmarty], [tantek], [tw2113_Slack_], [timothy_chambe], [snarfed], [KevinMarks], wagle, saptaks, [jacky], willnorris_, Guest4437, Xe, capjamesg, ancarda, strugee_, pharalia, rubenwardy, BinarySavior, prologic, oenone, Gorro_Rojo[theyi, Aidhan, JaeItIts[m], shreyanjain[m], oodani, jbrr[m], Soni, kushal, lockywolf, lanodan, nsh, bret, gxt__, rhiaro, holiday_1, oxtyped, neatnik, eb, alecjonathon, srushe, vikanezrimaya and jan6 joined the channel
# vladimyr Right, right
# vladimyr This is super weird... I'm using mastodon.social that is definitely aware of your existence and searching for
# vladimyr a) snarfed.com - gives nothing
# vladimyr Sorry s/com/org
# vladimyr Did correct search, just typed wrong domain here inside chat
# vladimyr b) your fed bridgy actor id - nothing
# vladimyr c) your @snarfed.org@snarfed.org - finally fetches your profile but shows link as unverified
# vladimyr What's interesting tho is if I refresh mastodon page after it shows your profile fetched by acct uri
# vladimyr Aka I'm looking at https://mastodon.social/@snarfed.org@snarfed.org
# vladimyr Hit refresh
# vladimyr Because of actor url I suppose
# vladimyr That's pretty weird handing of url from mastodon end, depending how you land on it you'll get different result 🙃
# vladimyr imagines folks coming from Twitter experiencing Heisen-urls for the first time :D
# vladimyr This gets better and better...
# vladimyr So I went and checked two akkoma servers social.kernel.org and nixnet.social
# vladimyr And from those you are totally unreachable
# vladimyr No matter what I input in that search field (homepage, actor id or acct uri) it gives nothing
# vladimyr Then I thought maybe that's because those instances don't know about you but they should still be able to pull your profile using webfinger
# [snarfed] thank you for all the testing. feel free to add details to https://github.com/snarfed/bridgy-fed/issues/374 and other issues, I have issues filed for a handful of implementations that don't have full interop yet
# vladimyr Now I need to find someones X Y personal mastodon instance that never heard about you that's still able at least find you through acct uri before I simply conclude that this whole thing is broken beyond repair 🙃
# vladimyr Tried that one
# vladimyr But our friend [snarfed] is fedi celebrity so posts reached that one too :D
# vladimyr I never thought I said this but now I actually need some of those defederated dumpster fires cause they shouldn't receive your posts
# Loqi 🗺🐉 Here be dragons is in general a warning of potential dangers or dangerous areas, originally on physical maps, on the IndieWeb it refers to particularly problematic silos that usually formed in response to entire sets of abusive behaviors and accounts being banned from Twitter https://indieweb.org/there_be_dragons
# vladimyr scores first victory today!
# vladimyr Could find you here but server doesn't know about any of your prior activity!
# vladimyr Bonus points for proving the theory without stepping in shittiest corners of fediverse
# vladimyr This is private instance of takahe (I hope I spelled that right) author https://fedi.aeracode.org/@andrew
# vladimyr Django powered multiuser masto alternative
# vladimyr This is just great /s https://ttm.sh/BZH.jpeg
IWSlackGateway and [snarfed] joined the channel
# [snarfed] Calckey, hmm, ok. https://github.com/snarfed/bridgy-fed/issues/531
# shreyanjain[m] Kinda sad that Mastodon's implementation of AP is sorta the de facto now
# vladimyr This is baffling, this should end up being simple webfinger request
[tantek] joined the channel
# [tantek] shreyanjain[m] it's what happens with /monoculture especially in the absence of a test suite
# vladimyr More visibility/searchability tales
# vladimyr So tchncs calckey instance procial.tchncs.de is unable to locate (obv previously unknown) https://fedi.aeracode.org/@andrew
# vladimyr But akkoma running on social.kernel.org returns search result for https://fedi.aeracode.org/@andrew
# vladimyr And doesn't show any posts meaning it wasn't previously aware of their existance
# vladimyr Now this is interesting
# vladimyr What makes akkoma able to pull Andrew's profile from their personal instance
# vladimyr And not [snarfed]'s from bridgy fed
# vladimyr And why calckey can't do both
# vladimyr Mind you akkoma was able to find profile both by actor id and acct: uri
# vladimyr This matrix of interop inconsistencies happily grows -.-
# Loqi It looks like we don't have a page for "Calckey" yet. Would you like to create it? (Or just say "Calckey is ____", a sentence describing the term)
# Loqi It looks like we don't have a page for "Akkoma" yet. Would you like to create it? (Or just say "Akkoma is ____", a sentence describing the term)
# [tantek] I just created a stub Lemmy page if that helps as a brief example: https://indieweb.org/Lemmy
# vladimyr (sure, once I grab proper keyboard 🙃)
# vladimyr Now I went to some random pleroma (what akkoma forked from) instance (stereophonic.space) and searched for Andrew and got this :O https://ttm.sh/BZa.jpeg
# vladimyr Pardon my French but what the flying duck is this, I gave it exact url!
# vladimyr Same thing with acct uri -.-
# vladimyr I believe it is safe to conclude that search in fediverse is beyond broken
# vladimyr Idea for fed.itoo.ls No2 - make working profile search
# vladimyr And to make things even worse who knows do all those described behaviours totally flip once you log into that instance...
# vladimyr And I'm really not into the mood of making dozens dummy fedi accounts...
# vladimyr goes outside to touch some grass
gRegor joined the channel
# [snarfed] accomplishment for the night, got verified links working in Mastodon! search for @snarfed.org@snarfed.org on https://indieweb.social/
Ludo joined the channel
# vladimyr Really, how? :O
# vladimyr What did you do?
# vladimyr Ok, tried from Simon Willison's instance and doesn't show as verified, tried from mastodon.social and still doesn't show as verified but indieweb.social works!
# vladimyr What kind of weird sorcery is this 😕
# vladimyr fosstodon.org - not verified
# vladimyr front-end.social - not verified
# vladimyr toot.cafe - not verified
# vladimyr m.webtoo.ls - not verified
# vladimyr Maybe it rechecks links on receiving new actpub activity 🤔
# vladimyr m.webtoo.ls - haven't seen you in a week
# vladimyr toot.cafe - boost from Jun 14
# vladimyr front-end.social - post from Jun 9
# vladimyr fosstodon.org - same as toot.cafe
# vladimyr mastodon.social - same as ^
# vladimyr indieweb.social - same as ^
# vladimyr So it's not activity based unless you sent some ap activity in the background 🤔
# vladimyr (also m.webtoo.ls - post from Jun 13)
# vladimyr Doesn't make any sense
# vladimyr Just taking indieweb.social and toot.cafe in comparison - shows same [snarfed]'s timeline, same mastodon version, only difference I can spot is one less follower reported on toot.cafe
sebsel joined the channel
# vladimyr Ok, more ways in which mastodon search is broken https://blog.maartenballiauw.be/post/2022/11/05/mastodon-own-donain-without-hosting-server.html#d57ed121
# vladimyr They simply copied webfinger jrd from their masto instance and statically hosted it under their domain
# vladimyr Which makes in turn results in:
# vladimyr > this approach works much like a catch-all e-mail address. @anything@yourdomain.com will match, unless you add a bit more scripting to only show a result for resources you want to be discoverable
# vladimyr But it works only if search is made by logged in user ⚠️
pharalia joined the channel
# vladimyr And then on top of that there is interesting case of Simon Willison who is searchable (even though I'm not logged in) https://ttm.sh/BLF.jpeg
# vladimyr (screenshot taken from mastodon.social instance
# vladimyr They host their own personal mastodon instance on fedi.simonwillison.net
# vladimyr So naturally when you search for @simon@simonwillison.net webfinger should ask simonwillison.net about their account and not fedi.simonwillison.net
# vladimyr But they proxied webfinger (together with nodeinfo and host-meta) back to their fedi.simonwillison.net instance: https://github.com/simonw/simonwillisonblog/commit/f112e57f8619852985f15a71c00309f5046b8f1a
# vladimyr This is their actor fetched from their mastodon instance https://ttm.sh/BLu.json
# vladimyr So they should be referenced either by @simon@simonwillison.net or https://fedi.simonwillison.net/@simon or https://fedi.simonwillison.net/users/simon
# vladimyr Where former is web url, and latter is actor id and hosts actpub doc describing it
# vladimyr And that disparity between acct uri domain and https actor url is possible due to some Mastodon configuration magic described here https://til.simonwillison.net/mastodon/custom-domain-mastodon#user-content-getting-a-vanity-address
# vladimyr Also when I said proxied that doesn't mean http redirect but internal s2s request and serving that back to the user
# vladimyr Which is essentially the same trick like https://blog.maartenballiauw.be/post/2022/11/05/mastodon-own-donain-without-hosting-server.html utilized
# vladimyr And Scott Hanselman also described in his post https://www.hanselman.com/blog/use-your-own-user-domain-for-mastodon-discoverability-with-the-webfinger-protocol-without-hosting-a-server
# vladimyr But as commenter SjoredV stated this seems to limited only to people who are logged in
gRegor, Guest4437, oenone, darylsun, tei_ and holiday_1 joined the channel
# vladimyr As usual I wasted great amount of time trying to reverse engineer things instead of reading the docs 🤦
# vladimyr Anyway, I finally figured out how Mastodon/Pleroma search works
# vladimyr Each time you do a search ajax request gets sent to /api/v2/search route of famous Mastodon client api
# vladimyr Pleroma/Akkoma copied Mastodon's so they do the exact same thing
# vladimyr But there is a catch
# vladimyr That route has 2 important params
# vladimyr 1) q=<what you typed into search box>
# vladimyr 2. resolve which is boolean defaulting to false if you are not logged into that instance
tei_ joined the channel
# vladimyr Resolve basically means use webfinger to fetch account data in case instance never encountered it before
# vladimyr And that param is only available if you are logged in
# vladimyr As documented here on Mastodon side https://docs.joinmastodon.org/methods/search/#v2
# vladimyr And here on Pleroma/Akkoma https://api.pleroma.social/#operation/SearchController.search2
# vladimyr i.e. if you search anonymously you are searching only that instance and whether it became aware of that account some time in the history
# vladimyr And if you are logged in your search extends to previously unknown frontiers via magic of webfinger
# vladimyr I'm not sure who thought that making search conditional based on logged in state is good idea but that's how it works
# vladimyr And that's probably how Misskey/Calckey does it too (didn't check it yet)
# vladimyr Also on topic of verified links
[KevinMarks] joined the channel
# [KevinMarks] I remember at some point there were complaints about how much overhead each webfinger call was taking so they put it behind login to reduce them
# vladimyr Essentially splitting search into two views limited local - anonymous and unlimited global - for logged in users
# vladimyr Which is sooo confusing for newcomers
# vladimyr Anyway back to original problem of link verification, here is what my anon search returned on mastodon.social for [snarfed]'s acct uri https://ttm.sh/BL_.jpeg
# vladimyr And now the same search made on indieweb.social https://ttm.sh/BLj.jpeg
# vladimyr So whatever you did in 5:50 UTC triggered verification services but only on that instance
# vladimyr What the hell, commit link gives me 404 :(
# vladimyr Ok this is commit https://github.com/snarfed/granary/commit/8b61d3b
# vladimyr ^ fixed link: https://github.com/snarfed/granary/commit/a037ba5
# vladimyr I still fail to understand what exactly prompted indie.social to refetch actor and redo link verification compared to mastodon.social for instance 🤔
# vladimyr Oh, I should've spotted this sooner https://github.com/mastodon/mastodon/blob/v4.1.2/app/models/account/field.rb#L77-L88
# vladimyr This is how mastodon side extracts url from actor field
# vladimyr And why [snarfed] had to change that html snippet
# vladimyr Specifically this genial (read it in French) condition on line 85 dictating that href prop must equal anchor's innerText 🤦
# [KevinMarks] WTF? So that will break lots of prefectly valid rel=me links
# [snarfed] still unfortunate, but I kind of get it. they're trying to make sure the human-visible profile link text is where the actual link goes, so that you can't mislead viewers by making it something different, eg <a href="https://bad.com/">http://google.com</a>
# vladimyr mornin o/ digs through akkoma code just in case...
gRegor joined the channel
# [snarfed] ^ if any BF users here (cc [tantek]) want to try, all you need to do is add a rel-me link to your site that points to https://fed.brid.gy/r/[HOMEPAGE] , eg https://fed.brid.gy/r/https://snarfed.org/
# [snarfed] ^ [aaronpk] looks like you already have that rel-me link to your actor's url https://aaronparecki.com/aaronpk . you just need to change the HTML in your actor's attachment[0].value to have the full URL inside the anchor text, eg <a href=\"https://aaronparecki.com\" rel=\"me\">https://aaronparecki.com</a>
# [snarfed] I have a todo to see how it handles eg `<a href="https://google.my-site.com">https://google.<span class="invisible">my-site.</span>com</a>` 😈
holiday_medley joined the channel
# vladimyr https://ttm.sh/BLL.jpeg
# vladimyr [snarfed]: there is your evil answer ^
# vladimyr aaronpk: lol 💯
# vladimyr Um where can I place my bet on the possibility of unicode homographic attacks also :P
# shreyanjain[m] [snarfed]: Oh, there's a fediverse matrix channel?
# vladimyr Since akkoma is obviously new to the verified links game (that PR I linked is 6mo olds) I had to do some proper digging to find profile with links attached but here it is https://ttm.sh/BL5.jpeg
# vladimyr And now view from the search side https://ttm.sh/BLG.jpeg
# vladimyr Nothing is verified but that's how akkoma formats links
# vladimyr I have no idea what's rel=ugc tho
# vladimyr Apparently stands for user generated content
# vladimyr Calckey has jsonfeed support! https://procial.tchncs.de/@benoegen.json
# vladimyr Just append dot json at the end
# vladimyr Calckey's link format https://ttm.sh/BLK.jpeg
[schmarty] joined the channel
# [schmarty] vladimyr++ wow lots of good interop research, here! may i offer a nudge that there is plenty here to add to the wiki and/or pull together into some posts for posterity? indieweb-dev is searchable but not a great archive.
# Loqi It looks like we don't have a page for "Calckey" yet. Would you like to create it? (Or just say "Calckey is ____", a sentence describing the term)
# vladimyr And finally how Calckey presents links through account search https://ttm.sh/BLz.jpeg
# vladimyr [schmarty]: Yeah I'm planning on documenting all this stuff
# vladimyr [tantek]: Calckey is new cool kid, fork of Misskey
# [tantek] Calckey is a fork of [[Misskey]], an [[ActivityPub]] server implementation that also supports [[JSON feed]].
# [tantek] [snarfed] I'm going to try this which I believe should work and be accessibility friendly (i.e. not create any noise for screenreaders) <a rel="me" href="https://fed.brid.gy/r/https://tantek.com/" aria-hidden="true"></a>
# vladimyr You can use link instead?
# [tantek] [snarfed] done, http://tantek.com updated
# vladimyr Someone logged into the mastodon instance should search for your profile to retrigger link verification on that instance
# [KevinMarks] mastodon now uses link headers itself, since it js;dr'd all it's html
jonnybarnes, boag, btrem, gRegor and gRegorLove_ joined the channel
# [snarfed] hmm yeah, gRegor I searched for you and [tantek] on indieweb.social, logged in, but neither https://indieweb.social/@tantek.com@tantek.com nor https://indieweb.social/@gregorlove.com@gregorlove.com have green checks yet
# [snarfed] https://indieweb.social/@aaronpk@aaronparecki.com doesn't either
[jacky] joined the channel
# [jacky] random question regarding IndieAuth and progressive enhancement: does it make more sense to use a refresh token to ask for more permissions versus asking for a whole new token (like invoking the flow all over)? I ask because I noticed that you can add extra parameters (like scopes) to the request according to the spec
# [snarfed] https://indieweb.social/@aaronpk@aaronparecki.com has the green checks now! no clue what changed
# [snarfed] the green check http://google.com on his profile is so good too
Xe joined the channel
# [snarfed] ok [tantek] gRegorLove, it seems like your rel-me links might need to point a different BF URL, https://fed.brid.gy/[DOMAIN] , eg https://fed.brid.gy/snarfed.org . feel free to update and try again. sorry for the runaround!
# Zegnat [jacky]: if you send scopes when using a refresh token to gain a new access token, the "requested scope MUST NOT include any scope not originally granted". So you can only go to fewer scopes, not add scopes. https://datatracker.ietf.org/doc/html/rfc6749#section-6
# [schmarty] This might make sense if an app wanted to _shrink_ its scopes. Not sure of an example case tho'.
gRegor joined the channel
# [snarfed] gRegor no luck yet. interestingly though https://indieweb.social/@gregorlove.com@gregorlove.com now at least shows BF's updated profile link, with the full URL in anchor text. but https://indieweb.social/@tantek.com@tantek.com doesn't. I've searched for you both logged in, not sure what's going on, whee
# gRegor Oh, is Mastodon checking the exact HTML value of the attachments on https://fed.brid.gy/gregorlove.com, including the invisible spans?
# [KevinMarks] Also, the verification is asynchronous, so you may trigger it by searching, but the verification runs in a queue later, and I think verification is lower priority than fetches for posts etc
# [snarfed] aha, figured it out. gRegor you're right, we needed the profile update, but also we had the rel-me link right the first time, https://fed.brid.gy/r/https://gregorlove.com/
# [snarfed] https://indieweb.social/@tantek.com@tantek.com now has the green check after both of those
btrem joined the channel
# [snarfed] Bridgy Fed green check docs are up: https://fed.brid.gy/docs#mastodon-link-verification
# [snarfed] hmm they say they support <link> https://docs.joinmastodon.org/user/profile/#verification
# [KevinMarks] They publish link now for cross mastodon verification
# [snarfed] aha I see green checks now on https://mastodon.art/@gregorlove.com@gregorlove.com and https://mas.to/@gregorlove.com@gregorlove.com (logged into both). gRegorLove we may now just be waiting for Mastodon caches to expire
# [snarfed] yeah on your user page, https://fed.brid.gy/web/gregorlove.com , but they're flaky right now
# [snarfed] yup, trying to work around an open bug in the wordpress webmention plugin 😢 https://github.com/pfefferle/wordpress-webmention/issues/359
Seirdy_ joined the channel
[0x3b0b] joined the channel
# [snarfed] gRegor got me, I felt like I tried a ton of stuff in https://github.com/snarfed/bridgy-fed/issues/493