#dev 2020-12-21

2020-12-21 UTC
[fluffy] joined the channel
#
[fluffy] I’m trying to debug an issue with WebSub subscriptions on feed-on-feeds, and the websub.rocks distribution test suite seems to only support h-feed. Is there any way to get it to provide atom or rss instead?
#
aaronpk websub.rocks was really only meant to test the websub subscription mechanism, not the content that is delivered over that mechanism. it'd be nice to expand it to more use cases but that's all it does for now
#
[fluffy] yeah, it’s just that FoF still only supports rss/atom and adding h-feed would be a rather enormous amount of work that I don’t want to deal with, and testing the websub subscription mechanism is precisely what I”m trying to do.
#
aaronpk which part are you testing? it has some support for atom/rss https://websub.rocks/subscriber
#
[fluffy] 300
#
[fluffy] subscribing works just fine, but actually receiving/processing updates not so much
#
[fluffy] which is to say it’s the distribution section I’m trying to get working
#
aaronpk ah yea that one could use some expanding
#
[fluffy] it used to work fine but it stopped working at some point and it’s a mystery as to when and why
#
[fluffy] I noticed it breaking when I moved to nginx but that shouldn’t have anything to do with it. I wonder if the `php://input` pseudofile works differently in nginx or something.
#
[fluffy] oh, hm, `$HTTP_RAW_POST_DATA` seems to be what I should be using.
#
[fluffy] maybe this is an easy fix
#
[snarfed] [fluffy] could you use https://granary.io/ to convert its h-feed to atom or rss?
#
[fluffy] oh, there’s an idea
#
[fluffy] although, will that also pass the websub updates through granary?
#
[fluffy] like I don’t suppose it has a translating websub proxy, does it?
#
lahacker snarfed is there another site that uses the same OAuth flow as Twitter?
#
lahacker i wonder how the conversation might have gone had i used another website
#
[snarfed] [fluffy] no granary doesn’t currently do websub
#
[snarfed] lahacker the same, this isn’t about twitter specifically
#
[snarfed] twitter is pretty much stock OAuth 1.1, with only very minor tweaks
#
[fluffy] [snarfed] yeah I’m specifically trying to debug something with websub
#
[snarfed] (twitter is very behind though, we’ve been on OAuth 2 forever)
#
[snarfed] [fluffy] right, granary would be just to get atom instead of mf2
#
[fluffy] okay so yeah it looks like nothing’s actually coming through from php for the raw POST data. weird.
#
[fluffy] I wonder if something changed in PHP 7.whatever
#
lahacker oh gee.. ok i swear this'll be the last thought experiment; i set the headless firefox to sync with the user's firefox account; no password, but still skirting the API; yay or nay? ;P
#
Loqi woot
#
[fluffy] … yep it was in fact removed in PHP 7. wtf.
#
[snarfed] lahacker we probably won’t be the best judges for straw men or splitting hairs 😁 hopefully we’ve communicated the spirit of the idea! which you can then take into consideration
#
[fluffy] oh so now you’re supposed to use file_get_contents(‘php://input’) which is what I was doing all along, still dunno why this isn’t working. argh
#
[snarfed] another argument is, maintaining scraping over time is a horrible arms race that constantly breaks, is painful to debug and fix, and you never win. i know, i’ve spent way too much time in it. APIs avoid that
#
[snarfed] (not entirely, but mostly)
[schmarty] joined the channel
#
[schmarty] oop came by just a bit too late fluffy: yep! reading `php://input` is the thing to grab raw POST body as far as i know.
#
aaronpk yea that's the thing about scraping, these sites make no promise on maintaining the html, whereas they do have a schedule for maintaining and deprecating API features, plus they actively fight against scraping
#
[schmarty] i do get confused by the delivery part of websub. is it required that the POST body contains the updated content? is it possible that it's only notifying of the fact that updated content can now be fetched? (not sure how it would even communicate that URL if POST body is empty. 🤔 )
#
[schmarty] definitely been too long since i looked at websub 😩
#
[fluffy] the subscription endpoint would maintain the notion of what the originating feed URL is
#
[fluffy] the POST body is supposed to contain the updated content but if it doesn’t contain any content the presumption is that the subscriber would then fetch the original URL
#
[schmarty] ah right. each subscription endpoint is unique to a given subscription URL.
#
[fluffy] the problem I’m running into is that superfeedr is sending a POST body with the updated RSS but for whatever erason FoF isn’t parsing it
#
aaronpk [schmarty]: HTTP Link headers
#
aaronpk but the subscriber is better off looking at the unique subscription URL to determine that anywya
#
[schmarty] aaronpk: Link headers is what i would have guessed! haha.
#
[fluffy] right, so like, when FoF makes a websub request it sets its delivery endpoint to e.g. example.com/fof/websub.php/123/aklfdjslfjskla
#
[fluffy] and so when the hub sends a ping it goes to websub.php and it parses out the feed ID (123) and a ‘secret’ (aklfdsetc.) which is just a very basic valildation thing
#
[fluffy] (I didn’t feel like implementing the actual signing mechanism)
#
[fluffy] and anyway then FoF correlates teh feed ID with the actual subscription
#
[schmarty] makes sense so far. sounds like maybe the publisher is not sending the updated content and therefore expecting the subscriber to fetch, as you described.
#
[fluffy] this is with superfeedr which is very much sending a fat ping, and a big part of what they do is only sending a delta of what changed
#
[fluffy] but now superfeedr seesm to have stopped sending out push notifications for changes to my feed entirely, probably because I hit some API limit or something. whatevs. I’ll just change FoF to treat all pings as thin pings.
#
[fluffy] I don’t even really care about websub or immediate updates, it just bugs me that this isn’t working
#
[schmarty] it's been so long since i set up superfeedr but this does trigger a memory that they aggressively dedupe? like they won't send out updates if the content hasn't changed. wouldn't surprise me too much if they put a timeout on it. 😩
craftyphotons__ and kitt joined the channel
#
[fluffy] they actually won’t send out updates for changed items either
#
[fluffy] they just send out new items
#
[fluffy] this is why at some point I want to make a module for publ that handles its own websub, webmention, and activitypub
#
[fluffy] well, less module and more companion, that’s publ-oriented but should be usable by things that aren’t publ too
#
[schmarty] haha gosh that certainly makes superfeedr useful only for increasingly specific use cases 😐
#
[fluffy] yeah
#
[fluffy] ugh I have so many projects I want to work on but so little energy for them
#
[schmarty] i strongly resonate with that frequency 😅
#
[fluffy] I recommend never developing a chronic pain condition
#
GWG Me too
#
GWG On the energy and so many projects
KartikPrabhu, sp1ff, ohthecode, MrHyde, blueyed, oenone, ShadowKyogre, treora, jeremycherfas, gRegorLove, [tantek] and alex11 joined the channel
#
@eniehack ↩️ webmentionに関する記事でそこそこ書いてたのでかなしい (twitter.com/_/status/1340933912973910017)
schmudde, petermolnar, [fluffy], [KevinMarks] and shoesNsocks joined the channel
#
@ikm Webmention - IndieWeb https://indieweb.org/Webmention (twitter.com/_/status/1341015788468105216)
shoesNsocks, [Raphael_Luckom], shoesNsocks1 and geoffo joined the channel
#
@zachleat ↩️ consider this an officially sanctioned webmention, ethan (twitter.com/_/status/1341042474148655105)
geoffo_, [tantek], [fluffy] and [schmarty] joined the channel
#
[tantek] Lol from that thread: https://twitter.com/zachleat/status/1341041573711245314
#
@zachleat contrary to industry best practices 700MB is too large for a web site (twitter.com/_/status/1341041573711245314)
nickodd joined the channel
#
@RobertWPearce TIL about WebMention from @theprincessxena: https://christine.website/blog/webmention-support-2020-12-02 (twitter.com/_/status/1341075260779732992)
#
GWG Asking again, does anyone support refbacks on their site?
#
[schmarty] GWG: i do not! but i find it fascinating!
#
beko What is refback
#
Loqi Refback is a linkback method that works using the standard HTTP Referer header https://indieweb.org/refback
gRegorLove joined the channel
#
beko Hm. Two pingbacks. Does that count?
#
GWG [schmarty]: I bumped up a plugin a few years ago and it was a bit noisy
#
GWG I want to make it simple
#
GWG What I did then was to have the webmention processing code try to render the refback, but that seems overkill
#
[schmarty] GWG: i occasionally think it would be interesting to use refbacks as a source for webmention discovery. i don't think i would collect and try to display them outright.
#
GWG [schmarty]: That's why I want to redo it
#
GWG I liked displaying them for the wiki, as it doesn't send webmentions
#
GWG Wonder if that's a project worth pursuing
#
[schmarty] i think it makes some sense. i am tempted to say that webmentions found this way should get a special flag, but, webmention as a spec already says that webmentions can be reported by anyone. 🤔
[Raphael_Luckom] and ciccarellome[m] joined the channel; nickodd left the channel
#
[tantek] using it as a source of webmentions (and flagging as such, good idea) seems reasonable, and then apply all your usual webmention processing logic
#
GWG [tantek]: The new webmentions version under continuous development has a new field called protocol which we might store it in
#
GWG pfefferle already uses it for incoming activitypub messages
#
[tantek] makes sense
#
jacky oh that would change the game for me tbh
#
jacky like lol building brides would be so much intuitive (instead of using heuritistics, it'd be just an explicit match)
#
jacky shoot, we could even have silos be defined as a pseudo protocol to make their incoming bits more native to the IndieWeb
#
jacky < brain explodes >
#
GWG jacky: Bridge all the things
jeremycherfas, [chrisaldrich] and [KevinMarks] joined the channel
#
[KevinMarks] How does protocol work?
[schmarty] joined the channel
#
GWG [KevinMarks]: It's just for sorting purposes, because all mentions are stored in the same table
jeremycherfas, willnorris, joe1, [Emma_Humphries], [fluffy], [snarfed] and skippy_ joined the channel
#
skippy_ is anyone doing anything interesting Indieweb-wise with DigitalOcean App Platform?
reed_ joined the channel