#dev 2018-05-03

2018-05-03 UTC
tantek, [tantek] and eli_oat joined the channel
#
GWG
Does anyone have a handle on the Google Maps API changes?
tantek, tomasparks, renem and KartikPrabhu joined the channel
#
tantek.com
edited /reacji (+590) "t using reply posts now for reacji, with in-stream reply-contexts! move display and sample markup from brainstorming to how to, why"
(view diff)
[jgmac1106], [chrisaldrich], tantek, leg, snarfed, chrod and eli_oat joined the channel; chrod left the channel
#
schmarty
tantek++ minimal reply contexts look great and are easily understandable!
#
Loqi
tantek has 31 karma in this channel (431 overall)
#
schmarty
the quotes around the names of github projects feels a little "programmy" to me, but that is a tiny detail.
eli_oat, leg, Kyle-K, jalcine, [kevinmarks], snarfed, cweiske, Strypey, KartikPrabhu, AngeloGladding, iasai_ and swentel joined the channel
#
Zegnat
Has anyone documented any brainstorms on URLs with fragments as permalinks? Example: https://grapefruit.zegnat.net/2018/04.html#dt201804141608Z
#
Loqi
Sebastiaan and I were on a philosophy bender, apparently. One of the concepts to come out of it is the head cache. When you have filed something in the back of your mind for future use. The problem, just like with a computer cache, is that ...
#
Zegnat
I am pondering how to extract the h-entry from such a permalink correctly. My first instinct is: run the mf2 parser on the whole page, find the object where the url property equals the URL of the page with matching fragment.
#
Zegnat
(If multiple such are found, check of uid === URL, else take the first.)
#
Zegnat
Basically copying the behaviour of finding a representative h-card for a URL: find the h-entry on this page that represents the given URL (which happens to contain a fragment identifier)
what joined the channel
#
Zegnat
I was sure there existed something about this already, but can’t seem to find it
jeremycherfas joined the channel
#
KartikPrabhu
Zegnat: I am sure whatever HTML parser you are using get the fragment for you already. Then you could parse that for mf2
#
Zegnat
That is problematic because you lose the context other algorithms are dependent upon: https://github.com/aaronpk/XRay/issues/69#issuecomment-386220154
#
Loqi
[Zegnat] This may require a lot more refactoring than I initially thought. It looks like, whenever a fragment URL is provided, XRay is only going to parse that little piece of HTML: https://github.com/aaronpk/XRay/blob/417cc1b3cc77ed86edccf72db174853ade1d9...
#
Zegnat
Yesterday I would have given the same answer, KartikPrabhu. But today I know more :(
#
jeremycherfas
I currently have `<link rel="authorization_endpoint" href="https://indieauth.com/auth">` in my HEAD. I guess I need to get rid of that, at least temporarily, to try selfauth?
#
Zegnat
You will want to change that href to point at your selfauth install
#
Zegnat
https://www.jeremycherfas.net/auth/ - or wherever you have put it
#
Zegnat
(Apparently not there, as that URL 404s)
#
jeremycherfas
Right. Wondering how best to test. Locally, and then open via ngrok, or in production. In some respects production is easier, even if slightly more dangerous.
#
jeremycherfas
Haven't started yet. Need breakfast!
Lukas1, AlsoLakas, [colinwalker], [kevinmarks] and loicm joined the channel
#
vanderven.se martijn
edited /selfauth (-102) "Remove outdated todo. Selfauth has no problem with authorization."
(view diff)
#
sebastiangreger.net
edited /GDPR (+1602) "some clarifications"
(view diff)
[jgmac1106], iasai_ and iasai joined the channel
#
skippy
i just recently switched from indieauth.com to selfauth. it's as easy as changing that header link, and then logging in from your apps again. Or at least, it was that easy for me.
#
Zegnat
It should always be that easy
iasai, [kevinmarks], barpthewire, [unoabraham] and [jgmac1106] joined the channel
#
schmarty
Zegnat: really interesting challenge about fragment permalinks and authorship! i put fragment IDs on posts on ghostparty.today and include invisible author data in each: https://ghostparty.today/#2018-02-17-150018
#
schmarty
but they are inside a #feed, and it would be neat for authorship to pick up the name / photo in the feed
leg, snarfed, davy__, tantek and [miklb] joined the channel
#
Zegnat
schmarty, yeah, trickery required. This is why I recommended against the use of fragments for permalinks at IWC Berlin, because I was pretty sure the tools aren’t ready yet. Much rather be the experimental case myself than someone who is just thinking about starting on the indieweb
sebsel, maingo, [davidmead], tantek, AngeloGladding, snarfed and jmac joined the channel
#
Loqi
[jmac] So the out-yonder website that linked to my blog which I was excited about yesterday, as it gave me a real-world non-Bridgy Webmention source to test? It's hosted by Tumblr, and therefore the URL that links to my site is http://t.umblr.com/redirect?b...
#
aaronpk
there is one part of the webmention spec that enables this, as long as the redirect is actually an http redirect
#
aaronpk
it also requires that your receiver accept any arbitrary URL in the first step of receiving a webmention
#
jmac
It's the second point that seems sticker.
#
aaronpk
so if you get a webmention to your webmention endpoint with a target URL of the t.umblr.com URL
#
aaronpk
and if you say okay cool i'll check it out
#
aaronpk
then you fetch that URL and follow all the redirects and end up with a URL on your website
#
aaronpk
then everything proceeds as normal
#
snarfed
i kinda appreciate that bridgy so dominates webmentions sent in the wild..but i'm kinda more saddened by it
#
snarfed
...but also in this case i suspect bridgy actually did send that wm, because tumblr doesn't send wms itself :P
#
aaronpk
micro.blog is getting up there now
#
aaronpk
but as long as bridgy is sending webmentions for likes it'll probably always win
#
snarfed
yeah exactly
#
sknebel
my endpoint does follow redirects, but from past discussions I remember that not many do
#
snarfed
jmac: what's the source of that wm? bridgy mostly handles the t.umblr redirects (it wraps all urls), but yeah, it technically has to send to the wrapped link, not to the final url, due to the wm spec
#
jmac
So the mention-sender would set the wm's source to (in this case) http://t.umblr.com/redirect?blah, and my receiver would go ahead and load it, even though it's not in my domain. And if the *ultimate* URL it ended at was one it did in fact care about, then it's a legit wm. Right?
#
aaronpk
yeah except target not source
#
sknebel
the target, not the source
#
jmac
RIght, right
#
jmac
The oddity here is the necessity of loading the target URL, which is at an unfamilliar domain, but we want to see if it'll redirect us to the domain we do accept wms for.
#
jmac
Is there a .rocks test for this
#
Zegnat
And rather than load it full (GET request), you probably just want to do a HEAD request, as you are only interested in resolving redirects
#
aaronpk
hm no i don't think so. I haven't done a thorough job of tests for receiving webmentions
#
jmac
Zegnat: Great point
#
jmac
snarfed: Not sure what you're asking. You want to know the real-world URL I have in mind here, with the tumblr-ized redirection link inside?
#
snarfed
jmac: the wm source. i expect bridgy sent the wm, i'm just curious
leg joined the channel
#
jmac
Oh, no, I don't think bridgy's involved here
#
aaronpk
i would be more surprised if tumblr sent the webmention
#
snarfed
ok! still.
#
snarfed
exactly. tumblr doesn't send wms :P
#
aaronpk
...yet
#
jmac
RIght okay. Nobody sent a webmention at all. :) The situation is that an arbitrary website on its own domain mentioned a post of mine.
#
jmac
Er, small-m mentioned, ha ha.
#
aaronpk
aha :)
#
snarfed
ahhhhhh ok
#
jmac
And I was thinking I could use it as a test case for manual webmentions. Then I saw that my current implementation wouldn't work, as written, because that page does not literally contain my own URL anywhere on it. The end
#
jmac
(They do in fact link to my post, but only through Tumblr's OOH YOU'RE LINKING TO AN EXTERNAL PAGE?! LET ME SEE WOW redirection service)
#
aaronpk
so yes if you got a webmention with that source URL and the target was your own URL, that webmention *should* fail validation since your URL is not actually present on the page
#
jmac
That's right.
#
Loqi
[antipolar] #609 Tumblr Rewriting URLs
#
jmac
So the solution, such as it is, is to have the target be that wacky Tumblr-redirect URL, and then have my webmention-processor go through the contortions described earlier.
#
jmac
I mean, that's doable
#
aaronpk
it means your webmention receiver is at slightly more risk of making unnecessary HTTP HEAD requests
#
jmac
And it's in the IndieWeb bailiwick of world-wide-web realpolitik, i daresay
#
jmac
Well, assuming I have done the right thing, the processor is running asynchronously from the receiver already. So having another avenue to receiving bogus webmentions for bound the garbage can isn't really much of a practical risk, yes?
#
aaronpk
yeah exactly. it just means you might be wasting some cycles verifying some spam
#
jmac
Hi, I run a website
#
Zegnat
And as checking against spam is a good thing, shouldn’t really be a problem
#
jmac
My apache logs are 98% zombies clawing at Wordpress exploits, by weight
#
jmac
I can handle a couple false webmentions
#
sknebel
many will do the target check synchronously, but you can do the "easy" checks sync and defer ones that need you to fetch external pages
#
aaronpk
well this kills one of the "easy" checks
#
aaronpk
which is that the target has to be on your own domain before you even fetch
#
jmac
Yes, that's true and unfortunate.
#
jmac
I guess I have to always send back an HTTP 201 for every well-formed webmention, in this case.
#
jmac
(Unless the target is on my domain, but a URL I don't recognize, somehow.)
#
snarfed
heh, we had somewhat this same conversation 2y ago in the comments starting around https://github.com/snarfed/bridgy/issues/609#issuecomment-173650187
#
Loqi
[aaronpk] This actually isn't as bad as it sounds. Right now, you can send webmentions with these URLs and still fall within the spec, but only because the spec doesn't specify criteria on whether a URL should be "[supported by the receiver](https://www.w3.org...
#
jmac
Out of curiosity, do we know if webmention.io or any other popular implementations implement this sort of possible-target-redirection check?
#
jmac
Or has this been "Well, if someone were to implement it, here's how'd they do it"?
#
aaronpk
webmention.io actually accepts webmentions on your account for any domain o.O
#
aaronpk
i'm not sure if it properly resolves the received URL or if it stores the pre-redirected URL
#
Zegnat
I accept any and all well formatted webmentions on my endpoint, I think
#
aaronpk
tests something
#
Zegnat
Feel free to send me garbage mentions to test my theory, haha
#
jmac
OK, good to hear
#
jmac
My work is cut out for me then. :)
#
aaronpk
aw apparently webmention.io doesn't resolve it properly
#
snarfed
unrelated, FB is now emailing FB app owners to remind them to resubmit for review. curious what people here who get them will do...
#
aaronpk
i just got that for my app ID I use with silopub
#
snarfed
i'm waiting a few weeks or months for bridgy, i want to hear some reports to see if people figure out their new criteria/process first
[jgmac1106] joined the channel
#
[jgmac1106]
[snarfed] Curious? Why are you sad brid.gy is popular.
#
aaronpk
diversity of tools is better than monoliths
#
aaronpk
one of the same reasons I am intentionally not opening up Aperture for signups. I want there to be more choices of microsub servers instead of everyone using the one I run.
#
Zegnat
Enough people here should be familiar with the NIH syndrom. So hopefully we’ll be seeing multiple solutions come up
[deeden] joined the channel
#
[deeden]
is slowly working on a sinatra-based microsub server
#
snarfed
also sknebel++ (iirc) for working on an inoreader version of https://snarfed.org/microsub-bridge
#
snarfed
[jgmac1106]: bridgy is a massive hack that only exists because silos don't send webmentions. ideally it shouldn't exist at all. :P
#
snarfed
and yeah to the diversity point, i'm also sad that there aren't any other meaningful backfeed implementations for the major silos. if i had put less work into bridgy, there might be!
#
aaronpk
at least ownyourswarm handles foursquare so bridgy doesn't have to
#
snarfed
aaronpk++ yeah that's pretty much the only other backfeed implementation i know of
#
Loqi
aaronpk has 141 karma in this channel (1628 overall)
tantek and [tantek] joined the channel
#
vanderven.se martijn
edited /rel-me (+327) "/* Articles */ Add article “The Real Deal About rel="me"”"
(view diff)
[eddie], KartikPrabhu, barpthewire, snarfed and gRegorLove joined the channel
#
tantek
thanks schmarty! now I just have to get around to packaging up and releasing the PHP functions I wrote for auto_url_summary (the human readable text synthesized from known structures of silo URLs) and is_one_emoji
snarfed joined the channel
#
tantek
will probably drop them into a file like cassis-lab.php since they haven't been tested across JS and I have no plans to use them clientside, though no objections to CASSISifying them especially if there is demand
#
aaronpk
hah i called the section in the indieauth chapter "sign-in workflow" not "login workflow"
#
aaronpk
didn't realize that
#
dgold
oh hells
#
dgold
I forgot I moved my blogs mention support over to wm.io/morris
#
dgold
I've spent an hour or two wondering why I wasn't seeing mentions feeding back from twitter/m.b
#
dgold
guess I need to get back on getting that all working :(
#
dgold
would love it if snarfed offered actual pagenums when looking at a feed's archive
#
snarfed
dgold: context?
#
dgold
ah, hey ryan
#
dgold
when looking at, say, my google+ `feed` of wm's in bridgy, the pagination offers `newer` & `older` links; this is fine
#
dgold
but it would be nicer (fvv) if it say showed that there are 12 pages of them
#
snarfed
nice but not easily scalable
#
snarfed
if it helps, the per page urls use timestamps for paging, so they're pretty permanent
#
dgold
thanks snarfed
#
dgold
schmarty: hey. My morris files all show up as empty
#
dgold
aha! morris uses file_get_contents("php://input");, but i'm using nginx.
#
aaronpk
that should work
#
schmarty
@dgold: oops! i thought php://input worked to read the body no matter what was fronting for php...
#
aaronpk
it should
#
aaronpk
i use that on nginx
#
schmarty
dgold: is morris creating files with no content?
#
dgold
schmarty: that's exactly what its doing
#
dgold
folder of mentions are all json files with 0 content
#
dgold
and the index is empty
#
dgold
refresh.php fails with: array_key_exists() expects parameter 2 to be array, null given
#
dgold
but that's probably because index.json is empty
#
jmac
Continuing from this morning's discussion webmentions and redirects: I've dug a little further into Tumblr's own redirection stuff, and if you request a redirection-service URL that it gives you, it returns (as HTTP 200) a tiny document with <meta http-equiv="refresh"> and javascript-based redirection. So, not HTTP-level redirection.
#
jmac
GIven that, should my webmention receiver roll its eyes and parse it and do the right thing, or give up?
#
aaronpk
oh yeah, i remember this now
#
aaronpk
that's ridiculous and i would not jump through hoops to accommodate it
#
jmac
fooey
#
Zegnat
Sounds like Twitter
#
aaronpk
twitter does http redirects for t.co
#
dgold
schmarty: just an FYI: It is now working properly
#
dgold
I have no idea what happened
#
dgold
oh - hang on, I had been sending the mentions through a symlink, I guess PHP doesn't do symlinks
#
www.boffosocko.com
edited /Posts_about_the_IndieWeb (+396) "Sebastian Greger article The Indieweb privacy challenge (Webmentions, silo backfeeds, and the GDPR)"
(view diff)
#
www.boffosocko.com
edited /videos_about_the_indieweb (+378) "General pencast discussion link and YouTube WordPress tutorial/walkthrough"
(view diff)
#
dgold
schmarty: ok - when a new mention is received, its wiping the content of everything
#
dgold
refresh.php `refills` them all - any ideas?
AngeloGladding joined the channel
#
gRegorLove
We ran into that t.co meta redirect with indiewebify.me too
#
snarfed
for t.umblr links, the destination url is in the z query param, just special case the domain and grab it from there
#
snarfed
(cc jmac)
#
gRegorLove
The rel-me library follows the meta refresh redirects if you want to check that. Thread: https://github.com/indieweb/indiewebify-me/issues/52#issuecomment-245544196 and PR https://github.com/indieweb/rel-me/pull/3
#
Loqi
[gRegorLove] Alright, guess I chased that rabbit trail more than was necessary. After testing the rel-me library locally, I learned that cURLing the t.co links is returning the proper redirects. So it was just a case-sensitivity issue after all. I verified the ...
#
schmarty
dgold: eep that sounds very broken! morris *should* be creating a new data/mentions/XX...XX.json file for each incoming webmention, then updating index.json to add the new mention to the list of mentions for the target page's path.
#
schmarty
it does rewrite index.json each time
#
dgold
it does create a new file, no problems there
#
dgold
its just that it, and _all_ the other XXXXX.json files are now empty, as is index.json
#
schmarty
that is so bizarre that all those files end up empty 😱
#
schmarty
it makes zero sense to me that it would empty out the *other* json files.
#
aaronpk
oh i think t.co returns an http redirect for certain user agents
#
schmarty
it really should only touch two files on each incoming mention
#
sknebel
Yeah, t.co cares about user agents. Was an issue with the relme browser extensions
#
dgold
schmarty: I'm all out of ideas, I'm afraid, I don't understand it at all
#
schmarty
dgold all that comes to mind right now is folder/file permissions and making sure that your php-fpm (i assume) process can write to them
#
schmarty
but if it's making and blanking files then it seems like it must have write access
#
dgold
index.php is creating (and emptying), but refresh.php works perfect.
#
dgold
I'm stumped.
#
loqi.me
created /Morris (+140) "prompted by gRegorLove and dfn added by schmarty"
(view diff)
#
martymcgui.re
edited /Morris (+53) "add link to project"
(view diff)
[kimberlyhirsh] joined the channel
#
schmarty
dgold: do you have a publish step set up in publish.php ?
#
dgold
no, I don't use publish.php at all
#
dgold
could that be the key?
#
schmarty
oh, it's quite possible, yeah!
#
schmarty
make an empty publish.php like: <?php function publish() { /* do nothing */ } ?>
#
schmarty
i don't guard against publish.php being missing entirely
#
schmarty
i would imagine PHP would throw an error because it's being included but missing and stop before doing any processing.
#
schmarty
but if there's no publish() function to call it should definitely be failing
#
schmarty
(none of that explains the empty files IMO, ...)
#
dgold
yeah, did that, nada
#
dgold
as in: all the files are again empty
[cleverdevil] joined the channel
#
[cleverdevil]
Hey, [schmarty], I did my demo last night at a meetup, and used screech as one of my sample apps. It went great!
#
Zegnat
schmarty, if you are depending on the publish file, make it a require rather than include? (Completely unsolicited programming advice, I just read a thing in the logs...)
#
schmarty
cleverdevil++ wow, cool!
#
Loqi
cleverdevil has 25 karma in this channel (82 overall)
#
dgold
i'll noodle more tomorrow, thanks schmarty
#
schmarty
Zegnat: it's a require_once, which adds to my confusion about why the program is doing other than throwing exceptions
#
Zegnat
Ah, yes, that should kill execution
#
[cleverdevil]
I also moved puny over to use MySQL for its backing storage, with JSON column types for MF2... works with RDS. - https://github.com/cleverdevil/puny/blob/master/puny/storage.py
#
schmarty
dgold: sorry i couldn't be more help today! this is baffling!
#
jmac
snarfed: thanks, yes. I hate the advice "just special-case the domain" for the usual reasons, but... it's tumblr, and I guess it's an acceptable trade-off to have 10 lines of code account for 800 pounds of gorilla
#
Zegnat
dgold, I am happy to have a look tomorrow too, if my timezone lines up better
#
schmarty
cleverdevil: MySQL for the entire storage? or just for an index? i was very interested in your flat-files plans. :}
#
[cleverdevil]
When a category is searched for, it uses MySQL to directly query for the category in the MF2 properties - https://github.com/cleverdevil/puny/blob/master/puny/storage.py#L125
#
[cleverdevil]
So, its for the entire storage. I got a version working with S3 only, but I had to create a ton of index files, and do a lot of round-tripping to get it all working. It ended up being super slow.
#
[cleverdevil]
Media uploads go to S3, still.
#
[cleverdevil]
But, MF2 content is indexed and stored in MySQL.
#
[cleverdevil]
S3 is tantalizingly close to being able to be a full-fledged JSON document index and store, but its *just* missing a few features. I expect that may change over time.
#
schmarty
pretty neat!
#
schmarty
cleverdevil: awesome video! it looks incredibly straightforward to publish services on lambda this way.
#
[cleverdevil]
Its pretty neat.
#
[cleverdevil]
Static file serving doesn't work, though, but that's OK, its better to push that content off to S3 anyway.
#
schmarty
it does make me wish for a video filter that collapses long sections of stillness
#
schmarty
similar to how audacity and other programs can collapse long silences
#
schmarty
ah, i was wondering about the missing logo :}
[manton], [chrisaldrich] and wagle joined the channel
#
Zegnat
skippy: if you want fail2ban (or something) to monitor failed login attempts on selfauth: https://github.com/Inklings-io/selfauth/commit/560c902368c3eafd2503d6642f1705d027ff4749
#
Zegnat
We just aren’t exposing it, because logging things to system logs unless you know what you are doing and have access to them yourself is generally not a good idea
#
skippy
i use https://wordpress.org/plugins/wp-fail2ban/ on a WP install; it writes to a local file rather than syslog.
#
Loqi
[Charles Lecklider] Description fail2ban is one of the simplest and most effective security measures you can implement to prevent brute-force password-guessing attacks. WP fail2ban logs all login attempts – including via XML-RPC, whether successful or not, to syslog ...
#
skippy
oh, no, i use *this* one:
#
Loqi
[Federico Rota] Description This plugin writes the log of failed access attempts (brute force attack) and invalids pingbacks requests ( by xmlrpc.php ). Very useful to process data via fail2ban. You can activate the log for each pingback request feature and stop t...
iasai joined the channel
#
skippy
and i use that to write to a custom file, rather than syslog.
[dshanske] joined the channel
#
Zegnat
Ah, alright, selfauth does not do writing to a custom file at this stage.
#
skippy
i run my own server, though, so i could read syslog. i just generally like to keep things a little more segregated. userspace stuff writing to syslog seems wrong.
#
skippy
how often do you see failed logins to selfauth? are script kiddies automating scans and attacking these endpoints?
#
aaronpk
assume anything with a password field is getting automated attacks
#
Zegnat
I am not logging it, so don’t know. This feature was proposed and implemented by ancarda. Presumably he is keeping track of it now.
#
Zegnat
logs as little as possible
#
[cleverdevil]
Grr... nic.io's website is busted and won't let me disable domain transfer lock 😛
#
skippy
i log more than i need. i keep meaning to remedy that.
#
skippy
but it has been fun to see fail2ban working on my WP install.
#
skippy
ponders how to add TOTP to selfauth, now ...
#
Zegnat
may have been working on an auth endpoint with TOTP
#
Zegnat
I just keep getting distracted by other projects
#
aaronpk
if you assume putting the TOTP key in the config file is acceptable then it should be quite easy
#
aaronpk
the hard part of TOTP is the UI for the setup step
#
Zegnat
I brewed up some insane system where I was encrypting the TOTP key using the password :P
#
Zegnat
So authentication was truly dependent on both inputs, and neither was really being stored on the server.
#
skippy
neat!
#
aaronpk
that's great
#
Zegnat
I at least haven’t come up with a reason why it would be less secure than any other way to store it.
#
Zegnat
So feel free to steal that idea :P
[kevinmarks] joined the channel
#
ancarda
skippy, Zegnat: I’d be very willing to write a more extensive logging platform (syslog, email, or log file). Possibly in a future version of selfauth?
#
ancarda
I only wrote that patch for my own needs (logging to syslog will trip up OSSEC, which will email me)
#
Loqi
[ancarda] #36 Logging via syslog
#
ancarda
skippy: I don’t use fail2ban, do you know if it can be configured to read from any file? Perhaps you could point it at /var/log/(messages|syslog) and include some kind of filter/parsing code to identify the IndieWeb lines?
chimo, tantek and loicm joined the channel
#
aaronpk
omg how many bridgy twitter apps did you make snarfed!
#
aaronpk
there are 15 authorized in my twitter!
#
KartikPrabhu
haha! I have 4
#
GWG
GDPR solutions in WordPress are surprising me
snarfed joined the channel
#
GWG
Comment anonymization functions, user data export...
renem and eli_oat joined the channel