#dev 2019-05-06

2019-05-06 UTC
eli_oat1, eli_oat, djmoch, [kenbauer], KartikPrabhu, chrisaldrich, gRegorLove, snarfed, [Rose], [calumryan] and gRegorLove_ joined the channel
#
[Rose]
!tell eddie, would you be open to a pull request to add colour theming to Indigenous? (I would love to be able to change that orange, and offering a few built in theme colours with a switcher and matching icons might be nice)
#
Loqi
Ok, I'll tell them that when I see them next
#
GWG
I think Eddie has been busy with life stuff. He'd probably love the help
gRegorLove and [pfefferle] joined the channel
#
[Rose]
I hope so! Just wanted to ask before I try to understand Swift too much and find out afterwards he's already done the work himself 😛
#
[Rose]
(There's quite a few changes I'd love to make actually, I should open issues for them)
cweiske, [Chris_Adams] and gRegorLove_ joined the channel
#
GWG
I have some... but mostly supporting things I already talked swentel into
swentel joined the channel
#
[Rose]
Well, I opened 2 issues, because it made sense to
[kevinmarks786], [jgmac1106], KartikPrabhu, [tonz], [frank], [pfefferle], [tantek], [calumryan], [Zegnat], jeremych_ and swentel joined the channel
#
[jgmac1106]
[tantek] from main channel...yeah I even thought about Fun little easter egg prizes...just needed an emoji mapped to each post type...really could work with any set of feeds not just microblog come to think of it...may not even need mapping if the unicode is linked on websites
#
[jgmac1106]
in a different note I need to figure out how to reset caching when I update the nav in my class websites. Students sometimes miss updates for long time
#
[jgmac1106]
but something I can try and fix over the summer
#
[jgmac1106]
have a good day all
[calumryan] joined the channel
#
[calumryan]
Yay! 66.0.4 update rolled-out and my Firefox extensions are back
#
Loqi
does a happy dance!
gRegorLove_, eli_oat, [eddie], [grantcodes] and snarfed joined the channel
#
Zegnat
[calumryan]: on what platform?
#
[calumryan]
macOS Mojave
[jgarber] joined the channel
#
[jgarber]
Hello, all! 👋
#
[jgarber]
Endpoint abuse/security question:
#
[jgarber]
Halting sending in that circumstance makes sense. Does similar logic apply to any of the other IndieWeb-ish endpoints (e.g. `authorization_endpoint`, `token_endpoint`, `redirect_uri`, `micropub`)?
#
[jgarber]
The Webmention spec outlines a circumstance where a malicious website might advertise a Webmention endpoint on `localhost`, `127.0.0.0/8`, etc. which could cause a sender to POST to itself.
#
[jgarber]
The IndieAuth spec outlines how a client should verify `redirect_uri`, but I couldn’t tell if any of the other endpoints had similar security concerns.
#
[jgarber]
Specifically, should https://rubygems.org/gems/indieweb-endpoints (or other similar endpoint discovery clients) broadly reject loopback URIs across all endpoint types?
#
Zegnat
It shouldn't have that problem, because you never have to fetch the redirect uri yourself. That's up to the UA
#
Zegnat
But will have to have a slightly longer think about it. E.g. you post to micropub endpoints the same way as you do to webmentions, so that could have the same issue.
#
[jgarber]
Right, validating acceptable `redirect_ui` values is on the client application. But this same logic from the Webmention spec _might_ apply:
#
[jgarber]
> When the sender discovers the receiver’s Webmention endpoint, there are few legitimate reasons for the endpoint to be localhost or any other loopback address.
#
[jgarber]
Maybe? I could be confusing some of the various behaviors.
#
Zegnat
No, I mean, you may not need to verify loopback because you are redirecting the UA to that URL. So it would be up to the UA not to load it if loopback is an issue.
#
Zegnat
But sometimes you want loopback there, e.g. when testing a local auth endpoint.
#
Zegnat
While your webmention sender probably does not want to start talking to it self. So there it makes sense to not use it.
#
[jgarber]
Ah, right right. So sounds accurate to limit the rejection of loopback addresses to Webmention endpoint discovery.
#
Zegnat
It may make sense for micropub/microsub clients too, but that is super client specific whether it matters.
#
[jgarber]
Thanks for your help, Zegnat!
#
Zegnat
No problem. Definitely keep thinking about it and open issues!
#
sknebel
these things can be !!!FUN!!!, and also depend on the scenario a lot
#
sknebel
(basic example: if I host a public thing from inside a protected network, and only have a hole for that thing poked into the firewall, someone being able to coerce it into accessing internal URLs and "helpfully" reporting error responses isn't wanted. If I run a bunch of e.g. known instances in a university network it'd be a shame if they didn't talk to each other because "internal network, not secure")
#
sknebel
there's the topic of problems with url parsers and fetching libraries too, e.g. I think if you use libcurl in some scenarios you can give it specially crafted urls that you can use to have mit make almost arbitrary network requests
#
sknebel
(and in many cases, a better solution is probably deeper in the stack, firewalls and such)
#
sknebel
(which you of course can't do everywhere)
#
sknebel
Zegnat: well, the Auth code verification is done by the client to a site-provided address
#
[jgarber]
For an endpoint discovery library, would it be useful to provide an option to pass in unacceptable URLs?
#
[jgarber]
Maybe something like:
#
[jgarber]
IndieWeb::Endpoints.get('https://sixtwothree.org', reject: ['localhost', /^127\.\d>{3}\.\d{3}\.\d{3}/])
#
Loqi
Jason Garber
#
[jgarber]
Where the `reject` option is an array of either strings or regular expressions matching host names that the discovery client should _not_ return in the result set.
#
sknebel
host names isn't enough
#
[jgarber]
Full-blown URLs would be more useful?
#
Zegnat
For the loopback thing you need to check DNS too. I do that in my PHP webmention denpoint discoverer
#
sknebel
if strict, that check has to happen at the exact point the hostname is resolved to an IP address with which the request is made
#
Zegnat
Mine can return the IP address it checked so I can force the request in the next step to go to the same IP (no chance for DNS poisoning between discovery and usage)
#
Zegnat
I need to get that thing merged and resolved so people see the latest version.
#
[jgarber]
Zegnat: My PHP is pretty weak. I have a sense of what this block is doing, but could you step through it? https://github.com/Zegnat/php-webmention-endpoint-discovery/blob/iwc-nyc/src/SecureEndpointDiscovery.php#L101-L113
#
[tantek]
I really like the "Local-first" framing of this article (I'm still cautious about CRDTs) - however all the local / user-model analyses in the article IMO apply to everything IndieWeb as well: https://www.inkandswitch.com/local-first.html
#
[tantek]
what is local first
#
Loqi
It looks like we don't have a page for "local first" yet. Would you like to create it? (Or just say "local first is ____", a sentence describing the term)
#
[tantek]
what is offline first
#
Loqi
offline first is a method of developing a website that once a user has visited it once on their device, it will always provide at least some content (posts) and or features (like creating posts) when they revisit even without internet access, using technologies like Service Workers and Local Storage https://indieweb.org/offline_first
#
[tantek]
close enough
#
[tantek]
offline first << 2019-04: [https://www.inkandswitch.com/local-first.html Local-first software] / You own your data, in spite of the cloud
#
Loqi
ok, I added "2019-04: [https://www.inkandswitch.com/local-first.html Local-first software] / You own your data, in spite of the cloud" to the "See Also" section of /offline_first https://indieweb.org/wiki/index.php?diff=60693&oldid=59035
KartikPrabhu joined the channel
#
[jgarber]
Zegnat: Disregard that walkthrough request. I RTFREADME. Thanks for documenting the `secureDiscover` bits!
#
Zegnat
If not clear, issues and or PRS related to readme are also super welcome, [jgarber] !
#
Zegnat
If you want a walkthrough of the code, happy to provide. But basically the first two it's are if the host itself is an IP, and the last else is to resolve DNS otherwise. After establishing IP, uses build in validator of PHP to make sure those IPs are acceptable.
#
[jgarber]
Thanks! I was able to divine the meaning of the constants with some clever searching. PHP’s got some handy utilities for matching IP patterns.
#
[jgarber]
> It is recommended that the Webmention sender uses this information for posting the mention. Use one of the checked IP addresses to post to (instead of the domain, which might redo the DNS lookup) with the host in a `Host` HTTP header.
#
[jgarber]
Zegnat: This is an interesting approach. I imagine the `Host` header would account for multiple domains being hosted on the same IP address…?
#
Zegnat
Yes. That's actually how that always works, because http requests only put the domain in the Host header, not in the request line
#
Zegnat
Sometimes it is a little tricky to do this though. E.g. curl still needs the domain based URL to make HTTPS work, so you need a separate option to tell curl not to resolve the domain name and define the resolved ip yourself.
#
Zegnat
Your own request lib may vary on that matter
#
aaronpk
where was that post about these issues?
#
aaronpk
or maybe it was a slide deck?
#
aaronpk
i didn't bookmark it apparently
#
Zegnat
Are you thinking of the URL parser differences, aaronpk ? That was a slide deck... I don't have it handy
#
aaronpk
the one that describes how to trick various HTTP clients based on the URL parser differences between curl and such
#
Zegnat
Sorry, can't easily clean up the URL on phone
#
[jgarber]
☝ There ya go.
#
aaronpk
google--
#
Loqi
google has -1 karma over the last year
#
aaronpk
that's the one
#
Zegnat
I haven't done any special protection against that. I looked at the DNS / IP issue, but it uses a PHP URL parser so the checked URL may suddenly point elsewhere when pit into your http lib...
#
sknebel
crap like that is why I think it makes some sense to capture that at the lower levels of the stack if you can
#
[jgarber]
Zegnat When POSTing to an IP address with the `Host` header set, how do you determine whether to POST to `http://<ip>/webmention_endpoint` or `https://<ip>/webmention_endpoint`…?
#
[jgarber]
POSTing to `https://…` would conceivably result in an SSL error if there’s no TLS certificate associated with the IP address…
#
Zegnat
You use the scheme that you discovered before. You don't change that.
#
Zegnat
And then see if your request lib supports it. Curl did not for me.
#
sknebel
(as in, if your webmention processing runs in an extra process and you're on your own server, you can firewall it from accessing any weird ports or local IPs)
#
[jgarber]
Okay, that’s what I suspected (re: scheme). Not sure what to make of possible TLS errors. Will have to play around with the HTTP library I’m using.
#
[jgarber]
Zegnat++ Thanks again for patiently answering my questions!
#
Loqi
Zegnat has 43 karma in this channel over the last year (146 in all channels)
#
Zegnat
In curl I needed to use --resolve and still post to the normal domain. Otherwise it wouldn't be able to verify certs
#
Zegnat
Basically you just want to look how you can pin or pre-resolve the domains IP. Mosy libraries will have some option for that.
#
[jgarber]
…and the intention of all of this is to make sure you are POSTing to the destination you believe you’re supposed to be POSTing to?
#
Zegnat
If your discoverer checked for loopbacks behind the domain, you don't want your http lib to resolve to a loopback IP in the next step.
#
Zegnat
That's why you want to pin it to the resolved ip that you have already checked.
dougbeal|mb1 joined the channel
#
[jgarber]
Phew, okay. I think I’m tracking most of this. 😄
#
Zegnat
Hehehe
#
aaronpk
this problem only happens if you are running some sort of sanitization on the URL before actually making the HTTP request
#
aaronpk
and it's a problem because URLs are hard to parse and your sanitization lib might be doing a different thing than your HTTP lib
#
sknebel
aaronpk: what's "this problem"?
[schmarty] joined the channel
#
aaronpk
the problem described in that presentation and being worked around in this discussion
#
sknebel
not quite
#
sknebel
these are two different things
#
aaronpk
maybe i wasn't following this convo close enough
#
Zegnat
My working around is purely about resolving the domain and DNS poisoning between DNS check and making the request. Has nothing to do with the URL parsers, and is instead about the loopback IPs section from the spec.
[Rose] joined the channel
#
aaronpk
that slide deck is relevant though because if you think you're checking for a loopback IP in your first step but your HTTP client is then parsing the input URL itself, you've introduced that problem described
gRegorLove_, [eddie] and [jgmac1106] joined the channel
#
Zegnat
Yes. But that is one I am not addressing (yet)
#
sknebel
sure, it's a weakness of that mechanism (potentially, depends on how exactly the fetch is done), but doing the check like Zegnat's code doesn't make it worse - it still eliminates some vectors
#
[jgarber]
Would it be possible to add any of these security-concerned pieces to the webmention.rocks test suite?
#
sknebel
difficult
#
Zegnat
The problem with the SSRF thing is that you need your request lib to expose functions for the URL parsing they use
#
Zegnat
And I don't know if they do. I can recall curl exposing that.
snarfed joined the channel
#
Zegnat
*can't recall
#
sknebel
afaik they recently announced a parsing API?
chrisaldrich, snarfed, [schmarty], jackjamieson and [jgarber] joined the channel
#
[jgarber]
> The Microsub endpoint URL MUST NOT contain a fragment, and MAY contain query string components.
#
[jgarber]
The Micropub and Webmention specs make no reference to endpoints with fragments. The IndieAuth spec mentions prohibiting fragments in URLs pursuant to profile URLs and client identifiers.
#
[jgarber]
Is it reasonable to assume that fragments are not permitted in _any_ IndieWeb endpoints?
#
sknebel
endpoint urls with fragments don't make sense, no
#
jacky
[jgarber]: good question!
#
jacky
yeah fragments / anchors are interesting for reading information but for sending a request? that seems a bit weird (aka not conventionally used in requests)
#
aaronpk
does HTTP say anything about POST requests and fragment URLs? cause that doesn't really make sense at all and isn't something we should need to define in every spec
#
sknebel
fragments are not even transmitted to the server
#
sknebel
HTTP on the wire does not know them
leg joined the channel
#
[jgarber]
I agree with the sentiment that a URL with a fragment isn’t super useful. Where it’s mentioned in the specs, it’s obvious that endpoints cann’t contain fragments which I infer to mean that a discovery client would flat out drop those URLs on the floor and return either no endpoint found or a different endpoint if multiple are specified.
#
[jgarber]
Is that the way y’all read that as well?
#
[jgarber]
A less strict interpretation could be, remove the fragment from the advertised URL and use the result as the endpoint.
#
[jgarber]
But also… there probably aren’t any examples of this behavior in the wild…?
#
jacky
^ that's it
[schmarty] joined the channel
#
[jgarber]
Oh, re: webmention.rocks and testing localhost/loopback => https://github.com/aaronpk/webmention.rocks/issues/20
#
Loqi
[aaronpk] #20 how to test avoiding sending webmentions to localhost?
snarfed and [eddie] joined the channel
#
Loqi
[OliverJAsh] #681 listen: support hash/fragment permalinks to entries inside feeds
#
sknebel
snarfed: the question was about the endpoints, not payloads
#
snarfed
ah. looked like a broader discussion, but ok!
#
snarfed
petermolnar: looks like the flickr likes you mentioned eventually got sent! https://brid.gy/flickr/36003160@N08/#responses
snarfed, [jgmac1106], [manton] and chrisaldrich joined the channel
#
GWG
[manton]: Have you been having problems with gigantic icons?
#
GWG
Coming from WordPress sites?
#
[manton]
@GWG Yes, the issue was WordPress using SVGs (icons like the location icon) without any width/height in the HTML. I added some CSS to parts of Micro.blog to make that icon small again. I personally think all img tags should have width/height when they are in feeds, because you can't rely on any default CSS when it's a feed.
#
[manton]
So, it's kind of halfway solved in Micro.blog. There are other assumptions Micro.blog makes about images, so it's not perfect yet.
#
GWG
[manton]: Rose brought it to my attention. It will be fixed
#
GWG
Also, they are inline, so no img
zoglesby joined the channel
#
GWG
[manton]: If I fix it on my side, you may not need to
[frank], snarfed, [filipe], [cleverdevil], KartikPrabhu, snarfed1 and [tantek] joined the channel
jcgregorio and jcgregorio_ joined the channel
#
@johnchidgey
↩️ Anyone that’s followed your work Marco would know this before asking. I have a similar opinion of WebMention/ActivityPub for blog entries, which is why I haven’t set that up either.
(twitter.com/_/status/1125549815792787456)
deathrow1 joined the channel