#dev 2021-05-31

2021-05-31 UTC
KartikPrabhu, wagle, shoesNsocks1, [jeremycherfas], geman, geman-, [KevinMarks], barnaby, jngldwf and [Shane_Gough] joined the channel
#
[Shane_Gough] Hello everybody. Just discovered the IndieWeb idea (and Wiki) and love the concept. I have a few questions though relating to the various protocols and I hope you can help. I apologise if I have missed some details in the various specs 😞
#
sknebel [Shane_Gough]: hi o/
#
[Shane_Gough] First off, the MicroPub protocol makes sense but the MicroSub protocol is described as a way to `consume and interact with feeds collected by a server` - how should I export the feed created by new entries created by MicroPub?
#
barnaby greetings [Shane_Gough]!
#
sknebel microsub is pretty much for a feed reader app to talk to a feed reader backend that collects feeds from various sites. the "indieweb way" for that is with microformats in the pages
#
[Shane_Gough] Thank you all 🙂 Very friendly group - more than most 🙂
#
sknebel e.g. if I post something to my site using micropub, the post on my website has microformats markup in the HTML
#
sknebel and if someones feedreader fetches my site, it can parse those microformats to get the details about the post
#
[Shane_Gough] [sknebel] I got that - but what is the best format to expose my feed? RSS, Atom? Is there a JSON format that is preferred?
#
barnaby [Shane_Gough]: the HTML, marked up with microformats 2, is how you make your posts machine-readable as well as human-readable
#
[Shane_Gough] Ahh, ok - so just the markup in the HTML is enough?
#
barnaby yep, exactly
#
[Shane_Gough] Ok, is there a preferred machine readable format though? In JSON or something?
#
barnaby the HTML + microformats *is* machine-readable
#
barnaby there’s a standardised mf2 JSON representation, but it’s not really intended for publishing
#
sknebel !mf2 https://www.svenknebel.de/posts/2021/5/2/
#
Kaja https://php.microformats.io/?url=https%3A%2F%2Fwww.svenknebel.de%2Fposts%2F2021%2F5%2F2%2F
#
sknebel post vs what a microformats2 parser extracts from that post
#
barnaby running HTML + mf2 through a microformats2 parser produces this standardised JSON format, which applications can then consume
#
barnaby some people also choose to use it as an internal storage or exchange format, but it’s not generally used to publish something to the public
#
[Shane_Gough] Ok, just thinking about things like malformed HTML etc. In Python or PHP you can probably process that fairly easily but in something like C# it's a bit of a pain 😞
#
barnaby hmm yeah I think we don’t have a C# mf2 parser yet https://microformats.org/wiki/microformats2#Parsers
#
aaronpk i would hope C# has good HTML parsing tools by now. malformed HTML isn't really a problem that anyone has had to think about in quite some time
#
aaronpk but generally consumers of the microformats data don't have to think about the HTML layer anyway, that's the job of the microformats parsing library
#
[Shane_Gough] I am sure it could be done but it would probably be messy 😞 Subcontracting that out to a separate service written in the appropriate language would probably be better.
#
barnaby [Shane_Gough]: I would assume that in any environment where you can run a C# app, it wouldn’t be too tricky to also install an mf2 parser in another language and call it from your C# code
#
barnaby or, if you feel like writing a C# parser, there are plenty of people here who could help out, at least with the parsing logic
#
[Shane_Gough] Yes, or have a separate parser service that just pushes JSON to a queue for processing by the C# service.
#
aaronpk i have done that kind of thing before for sure
#
[Shane_Gough] Or invoke a binary - whatever the architecture you are using is 🙂
#
barnaby hmm do we have an mf2 parser which is bundled up as a command-line tool already? I know the PHP one doesn’t do that, maybe the node or go parsers do?
#
barnaby looks like C# has some pretty good HTML parsing tools already! https://html-agility-pack.net/
#
[Shane_Gough] Ok, next question 🙂 What if the post (in HTML in RSS or Atom feed) cannot be turned into a valid mf2 structure? (is that correct - mf2 or do you prefer microformats2)?
#
barnaby mf2 is a good abbreviation! either is fine though
#
[Shane_Gough] If there are nice stable implementations that exist I would prefer to use them rather than re-invent the wheel 🙂
#
[Shane_Gough] I want to do *my* nice funky stuff - not build everyone elses again 🙂
#
[Shane_Gough] And probably badly 😛
#
barnaby exactly what is considered a “valid” mf2 structure depends on what point of view you’re looking at it from
#
[Shane_Gough] So the mf2 structures are still a bit flexible?
#
barnaby any working parser will be able to take even incorrectly marked-up HTML+mf2 and produce something a standardised mf2 JSON representation, it just likely won’t contain exactly the right content if the mf2 markup is messed up
#
barnaby but even with correctly marked-up content, consuming apps have to do their own validation and cleaning up in order to get something useful
#
barnaby e.g. applying the authorship algorithm to find the author of a post
#
barnaby as that may be embedded within the h-entry as an h-card, embedded as just a URL, or linked via rel-author
#
[Shane_Gough] Oh, that's not too bad. I mean worst case it includes the link to the source so you can go and have a look at that anyway.
#
barnaby I think https://indieweb.org/XRay is the best example of a service which performs these sorts of transformations https://indieweb.org/XRay
#
barnaby (also a good testing tool for seeing how your own mf2 markup gets interpreted
#
[Shane_Gough] That is a great resource, thanks.
#
[KevinMarks] also indiewebify.me gives a walkthrough of marking up your page
#
[Shane_Gough] What about visibility scope? I mean something like posts can be `public` (default), `private` (only visible to the author) or `restricted` (visible to a whitelist of subscribers). Those were just my initial thoughts - it seems that the assumption is that everything is public.
#
barnaby there has been some work done on private posts https://indieweb.org/private_posts
#
aaronpk that's something we've been working on slowly over the years, as always, anything requiring authentication opens up a whole can of worms
#
barnaby but yes, so far mostly the assumption has been that we’re dealing with public content, i.e. building off the pre-existing social infrastructure of blogs and personal sites
#
[Shane_Gough] Hah, I just realised the whole concept of `restricted` would never work 😞 If I allow you to see my post there is nothing stopping you from reposting it anyway so it may as well be public.
#
aaronpk technically yes, but also you could assume the people you share with will act in good faith and not repost stuff
#
barnaby well that’s more of a social problem anyway. it’s possible to e.g. follow a protected account on twitter, and screenshot or copy/paste content and re-post it
#
barnaby I’ve seen people do it in good faith by accident
#
[KevinMarks] google deck on competing app stores in china leaked via court case https://twitter.com/benedictevans/status/1399292996823269377?s=20
#
@benedictevans What happens when there are competing smartphone app stores? China is a case study. 700m+ Androids without Google services. Result: complexity… and higher commissions. https://pbs.twimg.com/media/E2tJhmLWYAA1jea.png (twitter.com/_/status/1399292996823269377)
#
[Shane_Gough] Yeah, it's more by accident than malicious intent is what I was thinking about. Both are possible of course.
#
barnaby as long as the ability to take screenshots exists, it’s not a problem which can be solved by technology
#
aaronpk the "on accident" part is definitely worth considering as we build tools around this
#
aaronpk e.g. a reader should probably not provide a "repost" button on a post that is visible to a limited audience
#
[Shane_Gough] So I'm thinking that it would be an implementation detail of the microblog itself - if you are logged in as the author you can see all posts made by that author, and you can mark posts as only visible by the author. If you are not the author then you only see public posts.
#
barnaby heh I found the screenshot I took of the last example I remember of a protected tweet getting retweeted publically, but now I’m questioning the ethics of putting it on the wiki as an example :/
#
aaronpk heh
#
barnaby maybe if I redact the content of the tweet
#
[Shane_Gough] I am just thinking that I have a 'real' (haha) job that might have me saving links as part of my diary that I either don't want to (or am not allowed to) share with others.
#
barnaby although there’s not really much left to look at then
#
[Shane_Gough] I think many people can think of similar examples 🙂
#
[Shane_Gough] Some might be a bit too personal 😛
#
barnaby I toyed briefly with having private posts on my website for things like that, but ended up deciding that my site isn’t really the place for them, and ended up storing data like that in local text files or my notes app
#
barnaby I know there are some people here who like to store private information on their personal sites though
#
[Shane_Gough] Yeah, what I really want is a shared UI, not so much a shared data source.
#
barnaby a shared UI for publishing, or consuming? or both?
#
[Shane_Gough] Both I guess - I am still trying to build up the perfect scenario in my head.
#
[Shane_Gough] So imagine this - you are building up a set of best practices to make docker containers, helm charts and local testing scripts to achieve a particular purpose for the company that is paying you money. A lot of the stuff you come across and the ideas you have are pubic (based on public examples) but the final implementation you do is private (refers to company internal policies, urls, etc). So you need to split them for security
#
[Shane_Gough] but for your own 'stream of consciousness` to figure out how you got from point A to point B you want to see everything.
#
[Shane_Gough] Does that make some kind of sense?
#
aaronpk i bookmark a lot of stuff on my site when i'm researching things, and i could definitely imagine posting some of the bookmarks privately
#
[Shane_Gough] Some of the bookmarks could be to internal company domains - or maybe not as internal as they like depending on the skill of the security team 😞
#
[Shane_Gough] The difference between 'work' and 'personal' is so blurred now 😞 I am already thinking of a MicroSub based reader that I could deploy inside the company I work for as a way to collect events from various internal systems and let people subscribe to the ones that interest them.
#
[Shane_Gough] I guess I am thinking about the whole stream thing as a lifelog - and there are different parts of your life that you share with different people. I guess `public` and `private` are enough - there are plenty of other channels available for `restricted`
#
aaronpk there's no reason not to try `restricted` too!
#
[Shane_Gough] Seems like an implementation specific feature (if you are logged in, and subscribed to my channel then you get to see my 'restricted' stuff and I trust you not to share it even by accident).
#
aaronpk the big question is how to handle "if logged in" across different websites when people are using different readers :)
#
[Shane_Gough] Well, what is the bearer token in the request?
#
aaronpk we've tried a few iterations on this but haven't yet found something that has "stuck"
#
[Shane_Gough] Oh, ok - sorry. I just clicked - if there is a bearer token it will be from the reader app right? Not the actual user of the reader app?
#
aaronpk if you're interested, start by reading up on https://indieweb.org/IndieAuth
#
aaronpk it would actually be a request from the microsub server since that's the one that is fetching the feeds
[schmarty] and [tw2113_Slack_] joined the channel
#
[Shane_Gough] Thanks, I will have a read
#
[Shane_Gough] I have to say - you guys are really great at linking to specific examples and documentation or chats 🙂
#
[Shane_Gough] I wish I could get the people inside the company I work for to do the same 🙂 Haha
#
[Shane_Gough] Ok, thank you everybody. I have enough to start to implement a microblog site that supports `private` posts I think.
shoesNsocks joined the channel
#
GWG I missed a great chat, darn
#
[Shane_Gough] Hmm, what tools exist for testing an authentication server (https://indieweb.org/authorization-endpoint#Auth_code_verification) running on localhost?
#
[Shane_Gough] I would like to test locally first before exposing on public domains.
#
aaronpk you could run one of the micropub clients locally too
#
[Shane_Gough] Oh, are there any that are purely web based? Like SPA style? I remember reading some of your blog posts linked from the Wiki but they were from 2015 or 2018 😞
#
aaronpk just cause they're old doesn't necessarily mean they are out of date!
#
aaronpk there should be some SPA ones if you want that https://indieweb.org/Micropub/Clients
#
[Shane_Gough] Hahaha 🙂 Fair enough 🙂 But JS has changed enough that things from 4 years ago don't actually work now 😞
#
Loqi [Shane_Gough]: lol
#
aaronpk i don't write about JS anyway :P
#
[Shane_Gough] Haha - at least JS has better backward compatibility than iOS or Android code so far
#
[Shane_Gough] Um, ok - I have to admit I have no idea how end user auth flow works 😞 I work on backend and infrastructure provisioning - so someone gives us an API token and we plug that into requests. When it breaks we ask for a new one.
#
aaronpk the guides on https://indieauth.net should help
#
aaronpk if you want a high level overview then this is good https://aaronparecki.com/2018/07/07/7/oauth-for-the-open-web
#
[Shane_Gough] From reading through this spec - https://indieweb.org/authorization-endpoint - it shows a page with permission (scope) requests and the serving page decides what to request to verify?
#
aaronpk i will also say, building an authorization server as your first step into the world of OAuth is going to be a challenge
#
[Shane_Gough] So if I decided to use Google or Facebook or Twitter as an authentication backend that wouldn't be a problem? I could ask them to do that on my page and when I got the response generate the `code` to send back?
#
aaronpk yes in OAuth it's totally fine to delegate authentication of the user to some other service
#
[Shane_Gough] But this isn't OAuth flow is it? In this case I could pop up a page that asks for a plain text password and generate whatever code I want?
#
aaronpk IndieAuth is an extension of OAuth
#
barnaby yeah AFAIK you can provide your own auth mechanic, which can totally just be a simple password, especially for a personal single-user micropub server
#
barnaby or sending a one-off email login link, or providing a challenge for the user to GPG-sign with their private key
#
[Shane_Gough] Or just saying 'yes' to any login?
#
aaronpk yes haha
#
[Shane_Gough] Ok, so this is where the trust comes in 🙂
#
[Shane_Gough] And also support for approved lists or ignore lists?
#
barnaby well if you’re building a single-user micropub server, then it’s likely only you who will ever be logging into it
#
barnaby so per-post permissions will be handled elsewhere
#
[Shane_Gough] But if I am building something for more than one user?
#
[Shane_Gough] Or if I was a complete a**h**e and decided to build a system that supported multiple IndieWeb users?
#
[Shane_Gough] Never mind, we are getting off topic. If you can recommend a link, chat, forum or something for security I would appreciate it 🙂
#
barnaby just to be clear, it’s possible to have a site which only acts as a micropub server for you, but which other people can log into e.g. for the purposes of looking at private posts
#
barnaby it’s only necessary to build a multi-user indieauth and micropub server if you want to make a service which multiple people can post to (correct me if I’m wrong aaronpk)
#
[Shane_Gough] Oh yes, I recognise that.
#
[Shane_Gough] But ideally, to be open, I should accept users from their own sites right?
#
barnaby what do you mean? users define which auth server to use with a rel=authorization_endpoint link on their homepage
#
@polarbirke ↩️ I‘m using „Comment“ button to Twitter reply with URL to Webmention myself. (twitter.com/_/status/1399431791413452807)
#
barnaby so when they log into your site, your site would look up their auth server of choice and start an auth flow there
#
[Shane_Gough] Maybe I don't have a complete understanding of the flow here - If you allow auth by domain and let me publish as that user I can push content to your site?
#
barnaby the flow can be a bit tricky to understand, because a personal site is often acting both as a client and a server at the same time. I definitely get confused about it a lot
#
aaronpk i'm pretty sure most people here don't let others post content to their site
#
barnaby you can log in to my site, but it won’t let you post there because it’s hard-coded to only let me post
#
aaronpk the main use of this flow is to authorize an application to post to your site when you are using that app
#
aaronpk so if you log in to https://quill.p3k.io for example, quill needs an access token from your site so that it can post to your site
#
[Shane_Gough] So I could set up a domain that auths everyone, login to your site as that user, and then push a bunch of rubbish until you block me?
#
aaronpk no, like i said before, most people don't allow arbitrary people to post to their site
#
aaronpk i think most people don't even let arbitrary people *log in* to their site in the first place
#
barnaby [Shane_Gough]: e.g. you can log in to https://waterpigs.co.uk/, and view my posting UI e.g. https://waterpigs.co.uk/notes/new, but it won’t let you post to my site, and if you log in with a micropub endpoint configured, my posting UI will create a post on *your* site!
#
barnaby the fact that arbitrary people can log into my site is left over from when I was experimenting with private posts
#
[Shane_Gough] I understand limiting posts to owners (or registered users of a site). But what about comments?
#
barnaby which people would have to log in to be able to see
#
barnaby [Shane_Gough]: an indieweb “comment” is a reply that you post on your own site, and then let the site you’re replying to know about it with a webmention
#
barnaby what is comment
#
Loqi A comment is a reply syndicated into the context of the original post https://indieweb.org/comment
#
aaronpk comments typically work a whole different way, using webmentions
#
[Shane_Gough] Ah, ok
#
aaronpk here's an overview post of how that flow works https://aaronparecki.com/2018/06/30/11/your-first-webmention
#
[Shane_Gough] Sorry everybody, I really am not trying to be contrary - I am just trying to get it sorted in my head 😞
#
aaronpk it's ok! there's a lot of moving parts
#
barnaby no worries!
#
Zegnat thinks there is only one site that lets anyone Micropub posts to it
#
barnaby you’re kinda jumping in the deep end by starting out implementing an indieauth server! might be worth starting out by working through the steps here to indiewebify your personal site and get familiar with how the most common indieweb building blocks work https://indiewebify.me/
#
[Shane_Gough] So, from what I understand the auth process returns a `code` which is basically a string? The consumer will then try to verify it by passing it back to the same URL?
#
[Shane_Gough] 555 - ok, I completely agree with that - jumping in with an auth server is probably the deep end 😞
#
barnaby Zegnat: which one is that?
#
Zegnat barnaby: https://sink.zegnat.net/
#
Zegnat Mostly for micropub client development testing, and was a sandbox for me to try out multi-author stuff.
#
Zegnat It is probably slowly aging away from spec compliance though, FYI
#
aaronpk this is also a good overview of indieauth https://aaronparecki.com/2021/04/13/26/indieauth
#
aaronpk it even has a more recent date :)
#
Zegnat barnaby: in case you are looking at test tools, there is also https://commentpara.de/ which can function as both an anonymous indieauth endpoint as well as a place to write webmention-powered comments without needing to use your own site.
#
[Shane_Gough] Oh my, if I insulted you about mentioning dates I am sorry 😞
#
[Shane_Gough] I can tell from the rest of your posts you had other things on your mind
#
aaronpk haha
#
[schmarty] [Shane_Gough] I can't speak for aaronpk but I doubt the dates mention was taken as insulting. that said, there _is_ a recurring theme that the constant churn and re-invention of web development tooling makes anything with a date on it increasingly seem "old".
#
aaronpk exactly :)
#
aaronpk i am just annoyed when people dismiss articles because they are 3 years "old"
#
[schmarty] haha oh the timing
#
sknebel maybe another point to have top-level dateless links for "important" posts like that
#
aaronpk that is a good point. i actually did that for my oauth post (from 2012) because it is *still* getting a ton of traffic
[dianoetic] joined the channel
#
[dianoetic] I took the dates off my list pages for just that reason aaronpk
#
[schmarty] indieweb building blocks tend to be developed through consensus as folks actually build them to interoperate. and they're typically scoped to solve specific problems, so a given spec (like IndieAuth) might "just work" for years between major changes.
#
[dianoetic] Still trying to find a balance between garden and "recent news"
#
aaronpk i will keep that in mind next time I do a "guide" style post like that
#
Zegnat I like what https://maggieappleton.com/garden/ is doing where posts will mature over time (seedling -> evergreen), which instantly makes me think of them differently versus normal time stamping.
#
@petergoes ↩️ Maybe something like: Netlify forms -> Webhook notification -> Netlify Function stores it somewhere (firebase?) -> fetch all on build? I did something similar with webmentions: https://www.petergoes.nl/blog/review-webmentions-before-publishing-with-github-actions/ (https://www.petergoes.nl/replies/2021-05-31-19-18/) (twitter.com/_/status/1399445513372684293)
KartikPrabhu, chee, jeremycherfas and [dianoetic] joined the channel