#dev 2021-06-18

2021-06-18 UTC
samwilson, tonz, nertzy__, [jeremycherfas], [tw2113_Slack_] and capjamesg joined the channel
#
capjamesg
What is hosting?
#
Loqi
Web hosting can be the primary regular cost in maintaining an IndieWeb site; this page lists several options from free on up depending on your publishing needs, like a static, shared, private, or dedicated server https://indieweb.org/hosting
#
capjamesg
Does anyone use git hooks for deployment?
hendursa1 joined the channel
#
Ruxton
capjamesg: kind of. i use them to trigger builds, which triggers deployment if builds pass
#
capjamesg
GitHub actions is great for deployment but not having access to a server is a big constraint.
#
capjamesg
I'd like to experiment with setting up my own webmention endpoint but I can't do that on github.
#
capjamesg
So I need to think of an easy method of deployment.
#
Ruxton
Actions != hooks
hendursaga joined the channel
#
capjamesg
Yeah, I know.
#
capjamesg
I'm just thinking about hooks if I move to a VPS.
#
Ruxton
with hooks + buildkite I can set builds to run on my laptop or some other connected computer, dont necessarily need a VPS
gerben, chenghiz_, tonz, oenone and barnaby joined the channel
#
jeremycherfas
!tell capjamesg Yes, I use a webhook to update my live Grav site when I update the repository.
#
Loqi
jeremycherfas: capjamesg left you a message 4 days ago: I finally published the blog post about converting my blog to .docx (but it's quite code heavy): https://jamesg.blog/2021/06/14/saving-blog-posts-in-docx
#
Loqi
Ok, I'll tell them that when I see them next
#
jeremycherfas
I saw that, and I like that it is code heavy and that you explain what the code is doing. Best way for me to learn.
#
mgdm
Tailscale had a cool post about using Github actions with ephemeral Tailscale keys to deploy things after a build https://tailscale.com/blog/2021-05-github-actions-and-tailscale/
#
mgdm
I'm not using tailscale myself but things like that keep tempting me
#
capjamesg
Thanks, jeremycherfas. That means a lot hearing someone say they can learn from my more technical posts posts :)
#
Loqi
capjamesg: jeremycherfas left you a message 7 minutes ago: Yes, I use a webhook to update my live Grav site when I update the repository.
#
capjamesg
Can you elaborate re: the webhook jeremycherfas?
#
capjamesg
mgdm thanks! I guess I'm looking for the most seamless method of deployment. One that doesn't require too much thought.
#
capjamesg
I used to build my Jekyll site on local and then FTP into the digitalocean server I used. But that's not efficient. GitHub actions has been great.
#
mgdm
I'm interested to hear what you come up with. Currently I have a VM and deploying is a manual `git pull && zola build`. I'd quite like Actions to just do the whole thing for me
#
mgdm
if I can spew markdown into Github's web interface and have it deploy to the VM that would be ideal for me
[Murray] joined the channel
#
[Murray]
How much back-end do you need? I'm using Netlify which autodeploys on push to GitHub, but also has a webhook that I can trigger directly. I've got buttons in my CMS and (I used to have) a button on a reader that I used that both trigger redeploys.
#
capjamesg
Well. This is where it gets interesting now I think about it.
#
capjamesg
I'd ideally like to run a flask server for webmention receiving / processing.
mikeputnam joined the channel
#
capjamesg
And I suppose I could serve all of the static files Jekyll builds from that.
#
mgdm
I could shift the whole thing to something like netflify except that I like having a VM to do odds and ends (like a Gemini server on the same hostname)
#
jeremycherfas
There is a little PHP script that pulls origin/master when triggered. There is no specific build process because Grav is dynamic, but heavily cached and served from markdown files.
#
[Murray]
yeah, hence the question of "how much back-end" 😄 (I have a server too, it's just decoupled from my site and runs stuff that needs PHP etc.). But for something like a webmention receiver/parser, that you could do in a serverless function via something like Netlify, or just via a Github Action. It depends if you want the mentions to render between builds or not?
#
capjamesg
Oh. Hey! I had never thought about serverless functions.
#
capjamesg
I suppose I could use a subdomain as a webmention endpoint. And have that hooked up elsewhere.
#
capjamesg
Because the webmention endpoint will not need much maintenance.
#
capjamesg
Wow. Setting up a webmention endpoint might be more difficult than I thought...
#
sknebel
because it's so tricky to pick how you want to do it? :D
#
capjamesg
There's that and I don't have much of a background in standards so I'll need to spend quite a bit of time reading / understanding the spce.
#
sknebel
the nice thing is that you can do it quite iteratively
#
sknebel
"receive and store requests" -> "validate requests" -> "extract data for display" -> "extract more data for display" ...
#
sknebel
(mine only does the first two and then notifies me, since I dont display webmentions yet)
#
capjamesg
Hm. Indeed.
#
capjamesg
What language are you using to receive / store requests?
#
capjamesg
I'm thinking about a flask web server for myself.
#
sknebel
python, yeah
#
sknebel
I think I cribbed a few pieces from bear's code on github, but I think mostly for sending
tonz joined the channel
#
capjamesg
Right. So, say I have implemented the validation. What is the content of the webmention? Should I be parsing the file w/ microformats and using the e-content h-entry attribute?
#
capjamesg
This might seem like a really obvious question. Again, I'm new to the spec.
#
sknebel
right. so the webmention spec really only discusses it up to that part, the rest is community convention
#
sknebel
and that's indeed parsing the post for microformats
[snarfed] joined the channel
#
sknebel
and then getting name/content/author/... and identifying how it linked to your post
#
sknebel
is it a reply to your post, is it a post that just mentioned your post somewhere, is it a like/bookmark/...
#
[snarfed]
many of us use libraries or services instead of implementing all this ourselves. https://indieweb.org/post-type-discovery#Implementations , https://indieweb.org/authorship-spec#Implementations
#
capjamesg
Right. And I can do that through searching for the microformat values like in-reply-to, etc.?
#
GWG
We really need to assemble all the stuff on presentation
#
capjamesg
I'm using mf2 atm for parsing.
#
capjamesg
Which makes things so much easier.
#
GWG
There are too many places on the wiki for it...A landing page would help
#
capjamesg
mf2util looks interesting.
#
capjamesg
Right. So..
#
capjamesg
Once I parse webmentions, I can save them to a file. And then I can write a REST endpoint to retrieve those files.
#
capjamesg
And that REST endpoint will have params to filter by URL so I can get webmentions by URL.
#
sknebel
for example
#
capjamesg
I'm understanding this now.
#
capjamesg
Thanks snarfed for pointing me to the mf2util lib. That will save a lot of time.
#
capjamesg
snarfed++
#
Loqi
snarfed has 23 karma in this channel over the last year (53 in all channels)
#
capjamesg
And sknebel for clarifying the spec.
#
capjamesg
sknebel++
#
Loqi
sknebel has 6 karma in this channel over the last year (32 in all channels)
#
capjamesg
What does this mean "The receiver SHOULD use per-media-type rules to determine whether the source document mentions the target URL."?
#
capjamesg
I'm not familiar with per-media-type rules.
#
capjamesg
I know I have to check whether source mentions the target. But what tags do I have to check?
#
barnaby
per-media-type means parse the source document according to its content-type
#
barnaby
e.g. as HTML+mf2 if it’s HTML
#
barnaby
in practise you’re only likely to need to parse HTML+mf2 content
#
aaronpk
That means for example if it's a JSON document look for whatever a "link" would be in JSON, like look for the url as a value of some property. It's not really wel defined for anything other than HTML but we didn't want to limit webmention to only HTML pages
#
barnaby
as for which elements, it’s up to you. I think most people just make sure that there’s a visible link somewhere on the page which matches the target URL
#
barnaby
that suffices for validation. Then when it comes to presentation, you might want to do additional parsing to determine the relation between the source and target, e.g. reply, like-of, mention etc
#
capjamesg
barnaby what's the best way to do that?
#
capjamesg
(find a visible link)
#
capjamesg
I could parse with bs4 and look for a link. Or I could just look for links with the u-in-reply-to class (which I don't think fits the spec though).
#
barnaby
parse the document as HTML, find all the <a> elements, make sure there’s one which exactly matches the target URL
[tw2113_Slack_] joined the channel
#
[snarfed]
at least, ignoring any fragment
#
capjamesg
Got it barnaby.
#
barnaby
true, [snarfed]
#
capjamesg
What does that meean snarfed?
#
[snarfed]
fragment aka hash is the optional part at the end of the URL after #
#
[snarfed]
btw before you get too far down the road of building something yourself, have you looked at all the ways other people with static web sites receive webmentions? https://www.google.com/search?q=static+site+webmentions
#
GWG
barnaby: The Webmention spec suggests per type parsing for verification, not just a
#
Loqi
[aarongustafson] jekyll-webmention_io: A Jekyll Plugin for rendering Webmentions via Webmention.io
#
barnaby
GWG: we were just discussing the per-type parsing requirement. are you aware of content types other than HTML which are actively used and should be parsed?
#
GWG
barnaby: Sorry, should do the full scroll down before commenting
#
barnaby
no worries :)
#
GWG
img, video, and audio tags come to mind
#
GWG
If you want to track people using your media
#
barnaby
yep, there are a few other legitimate places to look if you want to be really pedantic, like <area href="">
KartikPrabhu joined the channel
#
[snarfed]
bridgy doesn’t even parse the HTML, it just searches the raw HTML text for the URL. not recommended, arguably not compliant, but 😎
#
[snarfed]
but barnaby was asking about other _content types_, not just other parts of HTML. afaik we’ve seen discussion of non-HTML wms but no real attempts or experiments in the wild yet
#
GWG
[snarfed]: WordPress does that... I know I submitted a PR to change it to use DOMDocument and only use text as a fallback but I forget if we merged it
#
[snarfed]
huh. if you use text as a fallback, why parse at all?
#
GWG
I remember people talking about webmentioning json....
#
barnaby
[snarfed]: bridgy lives life on the edge ;)
#
[snarfed]
first we’d need an actual use case for non-HTML wms
#
GWG
[snarfed]: The parsed version is saved and passed to the response generator.
#
GWG
So it was being generated anyway
[KevinMarks] joined the channel
#
[KevinMarks]
svgshare.com accepts webmentions on svg and png files, but doesn't send them. Wasn't there a site that did it with JSON-LD too?
KartikPrabhu joined the channel
#
capjamesg
GWG so should I be checking for img, video, and audio too?
#
capjamesg
I don't really see that as something people would reference my content in.
#
capjamesg
Maybe images.
#
capjamesg
I have seen go-jamming. I read an article about processing webmentions which put me down this route snarfed.
#
Loqi
ok, I added "https://brainbaking.com/post/2021/05/beyond-webmention-io/" to the "See Also" section of /Webmention https://indieweb.org/wiki/index.php?diff=76120&oldid=76117
#
barnaby
capjamesg: the only case in which it makes sense to search embedded content for webmention links is if you’ve configured your web server to serve images, videos etc with an HTTP link header pointing to your webmention endpoint
#
capjamesg
Ah, I hadn't thought about that. That's very true. And something I have not set up.
#
barnaby
I’m not aware of anyone who does it. in theory it’s interesting for getting notifications of webmention-enabled sites which are hotlinking your media
#
sknebel
as [KevinMarks] said, svgshare does it
#
sknebel
Zegnat had made a list of what places in HTML take links: https://github.com/w3c/webmention/issues/91#issuecomment-334546716 :D
#
Loqi
[Zegnat] I think that would be worth mentioning in the spec then. From that table, it seems the following take URLs: | Attribute | Elements | | --- | --- | | `action` | `form` | | `cite` | `blockquote`, `del`, `ins`, `q` | | `data` | `object` | | `for...
#
sknebel
bad cell alignment Loqi
#
sknebel
(but yes, only a few are likely relevant in practice)
#
capjamesg
Is anyone comfortable reviewing a bit of Python code? :)
#
sknebel
sure (not right now, but in principle/later)
#
capjamesg
I've got something that technically works but I'd like a review to check implementation of spec.
#
GWG
capjamesg: If you utilize the information
#
capjamesg
GWG I'm not quite ready yet as I'm still ironing out a few issues. I'll open source my code when I'm happy with it so others can use it (as long as I get it compliant).
#
GWG
I never add complexity without reason
#
GWG
Although sometimes I don't end up using something I thought I would
#
capjamesg
Yep. I'm learning a lot from this.
#
capjamesg
But I've not implemented async task processing. I've never explored that in Flask before.
#
capjamesg
I need to sit on this for a little while. You make a good point re: adding complexity. But this has been fun!
#
capjamesg
How many people have their own receivers that they have built?
#
sknebel
I have a separate validation service that runs, the flask app just pushes things in an (on-disk) queue
#
capjamesg
Did you build the validation service :D
#
barnaby
I have my own. It’s 400 lines of PHP, a lot of which is boilerplate
#
barnaby
it doesn’t do as much advanced parsing or display as some people’s webmention handlers though (yet)
#
capjamesg
Ah, so I'm not the only one haha :)
#
sknebel
yes, I wrote that too
#
barnaby
it only knows how to parse mentions into likes, reposts, replies or plain mentions, and displays them all in chronological order rather than grouping them, e.g. https://waterpigs.co.uk/notes/5D9NcJ/
#
sknebel
(looked at the code again, lol, support for all kinds of weird things I forgot it did :D)
[jacky] joined the channel
#
sknebel
(e.g. it does one of the private webmention attempts, and does actually dump more data to storage than I thought it did. and has a terrible, terrible hack for supporting homepage webmentions. and it'll actually accept external urls that redirect to my site as target= parameter (which admittedly is a bit iffy))
#
barnaby
aaronpk: I’m currently running my IA server + Micropub adapter through micropub.rocks. Everything went fine so far, but on the GIF media endpoint upload test, the request fails and my server logs this error: “Header name must be an RFC 7230 compatible string.”
#
barnaby
looks like it’s seeing a load of headers made of whitespace
#
barnaby
any ideas? could just be an issue with one of the libraries I’m using
#
barnaby
the other two file uploads worked fine
#
aaronpk
Is it getting confused with the multipart request?
#
barnaby
could be, but all the other multipart requests worked fine, both the images to the media endpoint, and the multipart create requests
#
barnaby
it’s something about the GIF upload which is throwing it off
#
barnaby
Zegnat: you work on nyholm/psr7, right?
#
Zegnat
I sure do
#
aaronpk
I didn't think there was anything special about that other than it was a gif
#
barnaby
Zegnat: any ideas about why uploading a GIF might cause this stack trace? https://gist.github.com/barnabywalters/398b11fc183a71607f02b2f10069f706
#
barnaby
png and jpeg uploads handled by the same code path worked fine
#
Zegnat
barnaby: did you censor the header names or is that the exact strings? It seems like for some reason the HTTPS request has broken headers :/
#
barnaby
only thing I censored was my server path, with •••. everything else is exact
#
barnaby
very interesting
#
barnaby
can someone else who uses micropub.rocks for testing their https micropub server try out test 702?
#
Zegnat
Hmm, the only server implementation I run does not accept binary contents, so that would not be useful :(
#
Zegnat
It would be interesting to see if we can run just the one test against nyholm/psr7. I am happy to take a look this weekend and get the lib patched if the bug is there.
#
Zegnat
Sounds like it might be a problem in psr7-server though. That is the one trying to set empty headers. But it normally gets the headers straight from the server passing things to PHP, so, hmmm
#
barnaby
yeah it’s a weird one. could easily be something specific to my server configuration. I’ll see if I can make a minimal test case
#
barnaby
well, other than the GIF issue, and this https://github.com/aaronpk/micropub.rocks/issues/101, my micropub adapter passes all of micropub.rocks!
#
barnaby
so looks like it’ll be time for a 0.1 release soon
#
barnaby
100 lines of boilerplate/config/routing code, and 330 lines of adapter code
#
barnaby
I think that’s pretty respectable
#
barnaby
(the LOC counts are for the example micropub server which passes the tests, not the actual library)
tonz joined the channel
#
sknebel
left a comment on 101
#
barnaby
saw it, thanks
tonz, [KevinMarks], [schmarty], [tw2113_Slack_] and barnaby joined the channel
#
@TheGreenGreek
My talk on Webmentions + Eleventy @MagnoliaJSconf is up! I talk about the IndieWeb, what are webmentions, how to implement them, some cool stuff w/ GitHub actions and Netlify, and even include the shortest Eleventy demo ever made! https://www.youtube.com/watch?v=-R07ATpOh4M&t=98s https://sia.codes/posts/webmentions-eleventy-talk/
(twitter.com/_/status/1405987121043804169)
[jacky] joined the channel
#
[jacky]
what is vultr
#
Loqi
It looks like we don't have a page for "vultr" yet. Would you like to create it? (Or just say "vultr is ____", a sentence describing the term)
#
[jacky]
that's fine
#
sknebel
[jacky]: looking for opinions?
#
[jacky]
yeah tbh! I got my little test 'hosting' farm working with DigitalOcean and I'm thinking about adding other providers so there's options
#
[jacky]
Linode is next 👀
#
[jacky]
wow they're _very_ aggressive with their API
#
[jacky]
like you have to whitelist APIs to use
#
sknebel
haven't used vultr myself, but what I have in mind is "good quality, middling support"
#
barnaby
Zegnat, aaronpk: I experimented with the gif upload issues, looks like the guzzle HTTP client is the issue, not micropub.rocks or nyholm/psr7
#
Zegnat
Interesting. I think Nyholm might do some Guzzle work. We have had some Guzzle overlap with the psr7 implementation. So if you have a reproducable case, I would still be happy to have a look into fixing it :)
#
aaronpk
oh really! Maybe there's a fix for it already, it's been a while since I updated any dependencies on that
#
barnaby
aaronpk: I’m testing the upload locally using the latest version of the client, so I doubt it’s caused by you having an old version installed
#
[jacky]
oh I remember hearing that about their support
#
barnaby
as far as I can tell, it happens when uploading any gif
#
aaronpk
Oh huh
#
aaronpk
super strange
#
[jacky]
is this like uploading GIFs into Quill? tbh I thought it was my site and didn't bother reporting, lol
#
barnaby
[jacky]: could be related? my problem is with the micropub.rocks media endpoint GIF test causing mysterious server errors
#
barnaby
hm actually no, my bad, it’s not happening for all gifs
#
barnaby
seems to be happening reliably with animated gifs, not with static gifs
#
Zegnat
Interesting. But for sending files, Guzzle is just using PHP’s curl bindings, right? Do you get the same if you use curl directly? Or is that too hard to test?
#
barnaby
I haven’t tested curl by itself, but I have made some progress
#
barnaby
I replaced nyholm/psr7 with guzzlehttp/psr7 and the problem went away, so it’s a bug which happens specifically when using the guzzle client to upload an animated gif to a server using nyholm/psr7
#
barnaby
the mysterious whitespace header which guzzle/curl is sending shows up in both $_SERVER and apache_get_headers, but it looks like nyholm/psr7 is assuming that the headers returned from apache_get_headers() will be reliable
#
Zegnat
Just to clarify: are you using psr7-server to fill guzzlehttp/psr7, or did you swap that one out too?
#
Zegnat
We have had bug reports before for the wrong package, so just want to understand if nyholm/psr7-server is doing the wrong thing, or if nyholm/psr7 is doing the wrong thing.
#
barnaby
Zegnat: I’ve been experimenting in a minimal test case file where I only load the exact libraries I want to test. I’ve been comparing using psr7-server, and using guzzlehttp’s ServerRequest::fromGlobals() by itself
#
Zegnat
Gotcha
#
barnaby
it’s psr7-server which calls getallheaders/apache_get_headers, so that seems to be where the problem is
#
barnaby
although the problem is also clearly with guzzle client/curl adding these weird headers
#
barnaby
I’ll try the same upload using curl from the command line
#
Zegnat
psr7-server is a generic request to psr-7 implementation. So you could also use it to create instances of other psr-7 implementations. That is why I wanted to be more specific :)
#
Zegnat
As far as I know, getallheaders is the most reliable thing offered by PHP, so I would need to check how guzzle solves this then. Chances are they are just less fuzzy about following spec
#
barnaby
good point, maybe I’ll also test using psr7-server to make a guzzle/psr7 serverrequest, and see if the the problem still happens
#
barnaby
could be that guzzle are silently ignoring/validating the headers
#
barnaby
yeah, the root of the problem is clearly that the blank header exists at all, and whether or not server implementations should throw exceptions or ignore it silently is a different matter
#
Zegnat
Yeah. I will have to give it a little more thought. I do not think we will patch nyholm/psr7 to not throw. That one expects you to be building a correct and valid HTTP object. But I can see there possibly being a case for psr7-server being more lenient and just not pass on completely empty headers.
#
barnaby
okay, upload from command line curl doesn’t include the blank header! so looks like the guzzle client is the best place to file an issue, for now
doosboox joined the channel
#
Zegnat
Do tag me if you file an issue so I can subscribe to it ☺️
#
barnaby
writing it up now! will tag you
#
barnaby
“guzzle hates animated gifs” isn’t quite “openoffice doesn’t print on tuesdays”, but it’s definitely one of the weirder bugs I’ve found
[KevinMarks] joined the channel
#
[KevinMarks]
[capjamesg] if you want Python ones to look at there's my mention.tech (which uses appengine for async) https://github.com/kevinmarks/mentiontech/blob/master/main.py and Fluffy's https://github.com/PlaidWeb/Pushl/blob/main/pushl/webmentions.py
[aciccarello] and [schmarty] joined the channel