#dev 2016-09-02

2016-09-02 UTC
AngeloGladding and loicm__ joined the channel
#
kevinmarks.com
edited /framework (+61) "/* See Also */"
(view diff)
KevinMarks_, tantek and KevinMarks joined the channel
#
martymcgui.re
edited /bookmark (+59) "/* Silos */ instapaper acquired by pinterest (not pinboard, sadly)"
(view diff)
KevinMarks_ and tantek joined the channel
#
tantek.com
edited /clustering (+934) "stub with a definition, link to other pages / use-cases for clustering"
(view diff)
cweiske, KevinMarks and loicm__ joined the channel
#
cweiske
regarding my ES performance problems: I need to play around with keep-alive connections
#
cweiske
that might solve the problem
KevinMarks, loicm__ and cmal joined the channel
#
cweiske
ES performance: keep-alive did not change anything. replacing an does-document-exists query with a get-document query shrinked time for document with some hundred urls from 18s to 1.5s
#
cweiske
.. and with a HEAD query to get-document to 1s
#
cweiske
18x faster
#
cweiske
strange
#
cweiske
haha. HTTP_Request2's socket adapter is 1.5 times faster than the curl adapter
#
Loqi
hehe
#
cweiske
socket: 1s, curl: 1.5s
#
cweiske
.. now the only issue is that adding documents to ES also take 40ms minimum because of http://pear.php.net/bugs/bug.php?id=21108
#
cweiske
funny which bugs you can find once you dig into a project
#
cweiske
.. but the crawler is now in 2015 already, which is a huge improvement compared to the old code
rMdes, loicm__, cmal, KevinMarks and miklb joined the channel
#
martymcgui.re
edited /web_hosting (+66) "/* Virtual Private Server */ add myself to VPS users list"
(view diff)
cmal joined the channel
#
martymcgui.re
edited /User:Martymcgui.re (+2084) "Add "What Works" section. Update working on and itches."
(view diff)
#
martymcgui.re
edited /User:Martymcgui.re (+1770) "add wehavetoask.com, notes about personal domain."
(view diff)
cmal joined the channel
#
@DonaldBMcIntosh
@aaronpk Got indieauth working - yay! It was the http client, moved to httpcomponents. Also had to add your certs CA to my truststore.
(twitter.com/_/status/771726971730624516)
tantek joined the channel
#
tantek.com
edited /clustering (-6) "approach more than technique"
(view diff)
#
tantek.com
edited /MediaWiki:Sidebar (-71) "update for September"
(view diff)
#
tantek.com
edited /next-iwc (+4) "Brighton!"
(view diff)
#
tantek.com
edited /2016/Brighton (+110) "tent icon, minor copy edit"
(view diff)
cweiske joined the channel
#
cweiske
aaronpk, you here?
#
aaronpk
yep hi
cmal joined the channel
#
cweiske
chat.indieweb.org, config file: are channels given as "#chan" or as "chan"?
#
aaronpk
heh this config file got to be a little more than just config. i'll post a copy
#
cweiske
I've requested this already in an issue
#
cweiske
I had to manually add static functions to it to make it work
#
aaronpk
here's more or less the live "config" file
#
aaronpk
i'll update the one in the repo later
#
cweiske
so it's a full messy mix of "chan" and "#chan"
#
aaronpk
depends on whether it's used for URL matching or locating files on disk
#
aaronpk
the files have # in the filename
#
aaronpk
meh it's fine, i'm not really motivated to change it
#
aaronpk
interesting
#
aaronpk
that should work
#
aaronpk
btw in case it wasn't obvious, the fact that this uses a database is only a temporary measure right now
#
aaronpk
it's actually supposed to be reading the log files from disk, but I didn't finish migrating all the logs before I launched newloqi and this site
#
cweiske
I only work with what I have
#
aaronpk
yeah just wanted to warn you so you don't end up doing a lot of work with the DB since i'm going to rip all that out soon
#
aaronpk
oh weird. i thought i looked up the right format for that
#
aaronpk
thanks
#
cweiske
it was nearly correct but gave "UTC" when it wanted 0000
#
cweiske
begins to look better and better: http://indiechat.search.cweiske.de/?q=fail
#
cweiske
the one major issue is that join messages are not ignored but the full text of the page is shown in the result
#
KevinMarks
Did the log pages stop showing context?
#
cweiske
they did
#
cweiske
probably the db vs. logfile thing that aaronpk mentioned
#
aaronpk
yeah it's unfortunately harder to query for "nearby" entries with the file approach
#
tantek
really? they're not adjacent in the file?
#
aaronpk
they are. i didn't say it was impossible, just harder
#
aaronpk
with SQL it's super easy to be like "give me the previous 10 things"
#
aaronpk
but with the file, the previous 10 things might span to other files
#
cweiske
everyone stop now and document that as #1 reason against flat file storage!!
#
tantek
huh, I did implement that for my next/prev buttons in Falcon but I'm not sure what the trick was
#
tantek
even across file boundaries
#
aaronpk
finding the next 1 item is easier, but finding the next 10 can span up to 10 files
#
aaronpk
(super low traffic channels might have only one message a day)
#
tantek
sure. my "most recent n articles" code has to traverse backwards through files similarly
#
tantek
or rather, has to make that kind of request
#
tantek
the traverse backwards/forwards through storage files is common code
#
aaronpk
i'll do it eventually, but it's a lot of code i didn't have to write for the DB approach
#
aaronpk
i need that kind of seeking functionality for QuartzDB anyway so i'll only have to write it once
#
voxpelli
Had to write a similar thing against a Rest API that had a fixed upper count limit that was lower than what I needed
#
tantek
that's what I ended up doing, writing it once, and then depending on it in new and interesting ways with various features
#
tantek
alright, this is probably worth sharing openly here since it was just (yesterday released) OS X security update to patch a pretty bad vulnerability: https://support.apple.com/en-us/HT207130
#
aaronpk
i will merge/release that later but in the middle of something else right now
#
tantek
worth a restart
gRegorLove, miklb, tantek and cmal joined the channel
#
www.boffosocko.com
edited /annotation (+85) "W3C Open Annotation Community Group"
(view diff)
KevinMarks joined the channel
#
GWG
Good uncoordinated day.
KevinMarks joined the channel
#
tantek.com
edited /annotation (+145) "link to use-cases, separate News Genius page, silo vs other examples, specific Criticism subhead, link to WG, note CG was previous"
(view diff)
#
tantek
per the recent React discussion, note this blog implemented in React that works without JS even: http://jlongster.com/Presenting-The-Most-Over-Engineered-Blog-Ever
#
tantek
oh nevermind, he dropped React in his most recent rewrite! http://jlongster.com/RIP-Over-Engineered-Blog (pretty sure that's not a "joke" April fools post)
#
aaronpk
interesting post
#
aaronpk
"The last missing piece: one of the best features about dynamic blogs is the ability to drag-and-drop images right into a post." ... "I fixed this by hacking together a simple service which lets me drag-and-drop images and automatically upload them to my server. The site gives you back a URL and a markdown image tag ready to use."
#
aaronpk
that sounds similar to the idea of the micropub media endpoint
#
tantek
pretty strong validation of the concept
#
tantek
*independent validation
KevinMarks and tantek joined the channel
#
voxpelli
tantek: at work we build our site with React and it works fully without js (and actually doesn't yet or ever use React on frontend – not yet needed in any way)
rMdes and tantek joined the channel
#
tantek
snarfed, targets, sending, receiving, is this about /Webmention ?
#
Loqi
[Aaron Parecki] Webmention
snarfed and dmaczka joined the channel
#
snarfed
dmaczka sknebel: rejecting doesn't have to be synchronous. you can return a sync 202 and still reject later, out of band
#
tantek
re: redirect responsibilities, I think it tends to fall more on the receiver than the sender, EXCEPT for webmention endpoint discovery, which I think the sender MUST follow reidrects
#
tantek
one example is shortdomains. you may have multiple URLs for a post (short domain, long) and a sender may try to webmention any of them as the target
#
snarfed
also, on the receiver side, "check that target is a valid resource for which it can accept Webmentions" is generally expected to be internal. ie you're the receiving site, you know your pages, you can check whether a URL is for a page or for something else that would redirect without making an HTTP request to yourself
#
aaronpk
from my understanding of the question, it's also a matter of whether you want to accept webmentions for short URLs that you didn't create that point to your post
#
snarfed
ahh good point, that's a useful exception
#
sknebel
snarfed: sure, but if it is an external URL you can't tell internally
#
aaronpk
i'm not actually sure i'd *want* to accept those webmentions, but not 100% sure on that
#
dmaczka
tantek, yep, that's how this started, I was testing my webmention sender/receiver, had been using a nice happy source/target, then tried on a tweet I made linking back to a post, but of course twitter shortened my url
#
tantek
aaronpk's distinction is a good one
#
tantek
if they're not domains you control, I think you may reject
#
dmaczka
and so my nice-and-simple reject-all-hosts-I-don't-control rejected it
#
aaronpk
it's like someone says to you "hey I wrote a reply to your post, and also made my own alternate URL for your post here"
#
tantek
due to tco?
#
aaronpk
in this case yeah
#
tantek
can we punt on that until Twitter itself supports sending Webmentions?
#
aaronpk
tho the indieweb equivalent of that is if I were to wrap all links in my posts through my own short URL redirector
#
aaronpk
which incidentally I used to do on my website pre-2003
#
snarfed
dmaczka: so yeah, one alternative is to return 202, verify async, and follow target redirects
#
snarfed
tantek: not really, lots of people and sites generate and use short urls, not just twitter
#
aaronpk
so that i could check which outgoing links people were clicking on
#
snarfed
eg tumblr now wraps all outbound links too :(
#
dmaczka
I think I will do that for now... because right now will something like brid.gy send webmentions on urls in tweets?
#
dmaczka
which I'd want to catch
#
tantek
snarfed, is this primarily a silo problem then? Twitter, Tumblr, FB?
#
snarfed
dmaczka: yes but bridgy is careful to follow (unwrap) all urls to be nice to receivers
#
tantek
I'm more interested in the indie to indie case
#
aaronpk
tantek: probably primarily, although i just gave you an indieweb example that I actually did!
#
tantek
where the source post uses its own URL shortener for all outbound links
#
snarfed
yeah, primarily but not solely. people manually link with short urls too. just not globally like twitter, tumblr, some wp.com, etc
#
dmaczka
snarfed: but doesn't that negate the reason for including target in the webmention in the first place: to make it easy for receiver to scan the source for it?
#
tantek
so the question is how does a receiver verify that?
#
tantek
dmaczka: exactly
#
aaronpk
dmaczka: the source URL parameter still has to be in the source HTML as an exact match
#
snarfed
tantek: like i mentioned, you 202 and follow target redirects async
#
aaronpk
er, target URL parameter in the source HTML
#
snarfed
definitely optional
#
snarfed
i doubt many current wm receivers do it
#
tantek
I think there is a "respect hyperlinking and giving others search-juice" value we could use here
#
dmaczka
well, the one I'm making will:)
#
tantek
that is, if a site is "up" enough for you to be sending a webmention to, then you really should be openly linking directly to it
#
snarfed
yup that's the beauty of making it optional in the spec, with maybe an opinionated recommendation
#
aaronpk
yeah deciding whether a target URL is "a valid resource for which it can accept webmentions" is up to you
#
tantek
the only reason I'm considering making my own short URL wrappers for sites I link to is if I don't trust them to stay up - usually silos
#
aaronpk
eh that's not totally fair
#
snarfed
tantek: but even those silos can accept wms sometimes
#
aaronpk
i'm pretty sure more of my outgoing links to indieweb sites are now gone
#
tantek
snarfed really?
#
snarfed
tumblr, blogger, wp.com via bridgy
#
snarfed
or via wm.io
#
tantek
a-ha - some silos allow you to edit your templates sufficiently to provide webmention endpoint discovery?
#
aaronpk
yep and also wow i forgot how powerful the discovery step of webmention is
#
sknebel
since you don't actually have to fetch and parse the targer-URL, but just do a HEAD requests (and follow 30x codes) the work is not as bad
#
snarfed
tantek: sure, that's the whole point of bridgy for blogs, which is years old now :P https://brid.gy/about#blogs
#
tantek
skenebel I doubt any of those silos allow you add LINK HTTP headers to HEAD requests
#
tantek
happy to be proven wrong
#
aaronpk
tantek: no that's about following the e.g. t.co redirect
#
sknebel
tantek: sorry, wrong context. I meant for just checking if the URL redirects to my page at the end
#
snarfed
i actually doubt *any* current wm receivers follow redirects on target urls. bridgy for blogs doesn't.
#
tantek
ok. but the sender still has to set the *target* to the literal link that is in the *source* right?
#
aaronpk
it's like if I send a webmention that looks like source=aaronpk.com/foo target=t.co/12345 where t.co/12345 actually redirects to tantek.com/foo, I still need to have t.co/12345 in my post.
#
tantek
so for a tweet source, you'd have to set the target to the tco URL
#
snarfed
wordpress's webmention plugin doesn't (follow target redirects). iirc known doesn't.
#
aaronpk
but the receiver would need to check if t.co/12345 redirects to a URL on tantek.com in order to know what post it's for
#
snarfed
dmaczka could be the first!
#
sknebel
mine does I think
#
snarfed
ooh ok
#
tantek
so do we require receivers to follow redirects on the 'target' param before checking to see if they handle it?
#
tantek
or can they do a simple domain prefix check on the target URL?
#
aaronpk
it's up to the receiver right now
#
snarfed
sknebel: you should mention that in http://indieweb.org/webmention#IndieWeb_Examples !
#
GWG
snarfed, I changed the redirects following in WordPress
#
tantek
yeah totally
#
snarfed
GWG: you're thinking about sending wms? this is for receiving them, and checking the target url
#
tantek
aaronpk: does webmention.io verify the target pre-redirect, and then follow redirects on the target and provide that final target destination?
#
sknebel
the wiki FAQ says it's required, but links a non-existing section and not the spec: http://indieweb.org/Webmention-faq#Why_is_the_target_URL_a_required_parameter
#
snarfed
looking at the current repo head, the wp plugin definitely doesn't follow target redirects when receiving
#
aaronpk
checking webmention.io...
#
aaronpk
it actually doesn't verify the target at all
#
snarfed
makes sense
#
aaronpk
because it's meant to support receiving for multiple domains on your same account
#
aaronpk
but that actually probably means following redirects on the target is more important
#
aaronpk
because right now there are likely "orphaned" webmentions sitting there
#
tantek
yeah - that's what I was wondering - does it provide both pre and post following redirects on the target
#
dmaczka
another complication... so say the target is a t.co/foo... does my receiving server save that, or the expanded target uri for the purposes of later detecting when a webmention is re-sent?
#
dmaczka
to e.g. update it
#
tantek
dmaczka: I don't think you should have to save either?
#
snarfed
sure you do, for handling updates or deletes
#
aaronpk
updating is about the source URL
#
snarfed
oh target, nm
#
tantek
once the receiver determines the target, they check to see if they already have some record of the *source*
#
dmaczka
ok
#
tantek
there's no reason to save the target from a webmention AFAIK
#
tantek
says he who hasn't yet implemented a webmention receiver :P
#
aaronpk
speaking of which, just checked tantek's account in webmention.io
#
aaronpk
it has received webmentions with a target domain of: tantek.com, ttk.me, t.co and snarfed.org
#
tantek
😳
#
aaronpk
somehow this ended up getting sent to tantek's webmention endpoint https://snarfed.org/2014-03-26_8628
#
Loqi
[Ryan Barrett] likes Kyle Mahan: A reply from 2014-03-26.
#
snarfed
aaronpk: ooh how many total? and of those, how many are from twitter to his homepage? :P
#
snarfed
ie what's the @t mention count :P
#
snarfed
heh, that snarfed.org post is a timeless classic. one of my all time faves.
#
aaronpk
hm looks like the t.co didn't end up with any verified webmentions on it
#
snarfed
hey btw GWG while you're here, any idea why my WP site isn't sending webmentions? it's using the wm plugin at git repo head. hasn't sent them for months now. nothing relevant in the debug log. :(
#
tantek
snarfed, any idea why Kylewm's comment on https://snarfed.org/2014-03-25_bridgy-publish shows up as one line "pre" text? (have to scroll to the right to see the whole thing)
#
Loqi
[Ryan Barrett] Bridgy Publish
#
www.boffosocko.com
edited /Template:WordPress (+165) "Added group of pre-existing plugin pages"
(view diff)
#
aaronpk
wow webmention.io has received 64106 home page webmentions for tantek
#
tantek
😂
#
tantek
is really not looking forward to debugging those
#
tantek
Bridgy++ for good problems to have
#
Loqi
bridgy has 49 karma (1 in this channel)
#
tantek
is anyone else seeing the one-line-pre treatment of the comment from kylewm on https://snarfed.org/2014-03-25_bridgy-publish
#
Loqi
[Ryan Barrett] Bridgy Publish
#
snarfed
tantek: yes
#
tantek
just verified in FF and Safari
#
snarfed
the wp wm receiving plugin(s) had some teething troubles for a while as they learned to handle mf2 post types
#
snarfed
around that time
#
snarfed
i can go fix manually, just not a high priority
#
www.boffosocko.com
edited /Semantic_Linkbacks (+0) "WordPress template"
(view diff)
#
tantek
snarfed, just curious if this is a plugin defaults issue, or something to do with how kylewm marked up his reply post or ... ?
#
tantek
I have no doubt you can fix manually ;)
#
snarfed
it's an old artifact from bugs/missing features at that time
#
sknebel
could you just resend the mention and have it fix itself?
#
snarfed
sknebel: definitely! just hasn't been a priority. this conversation here is more than i've thought about it in years :P
#
sknebel
now I wonder if there is value in randomly re-evaluating old WMs every now and then (like, one every night or every few hours) to automate that and clean up dead links
#
sknebel
(which would be a reason to store the target-URL)
#
www.boffosocko.com
edited /Post_Kinds_Plugin (+31) "WordPress template & see also link"
(view diff)
#
snarfed
sknebel: what would you do when you find a dead link?
#
sknebel
snarfed: good question, not sure what my personal preference would be
#
sknebel
either treat it as delete (but on the other hand our convention is explicit delete with 410, but not all external sites do that of course), or hide the mention?
#
sknebel
maybe the idea isn't very useful ;)
#
GWG
I am thinking about receiving them
#
GWG
snarfed, this is why we need better de debugging and unit tests.
#
snarfed
GWG: context? re my site not sending wms?
#
GWG
snarfed, I had the same problem. I thought it was a cron issue
#
GWG
I have a few ideas for debugging improvements.
#
snarfed
how did you fix it?
#
GWG
snarfed, I didn't
#
snarfed
ugh. so your site isn't sending outbound wms right now either?
#
GWG
I wanted to try and figure it out.
#
GWG
It is why I changed so there is a hook for logging outbound
#
GWG
snarfed, give me some time. I am trying to reimplement Webmentions.
#
snarfed
that seems...ambitious...at best
#
GWG
Using the REST API
#
GWG
snarfed, how so?
#
GWG
I want an implementation that might get into Core.
#
snarfed
ah, ok.
#
snarfed
we have different goals then, i'd just like this bug fixed :P
#
GWG
snarfed, I am multitasking.
#
GWG
But either way, I intend to figure it out. I am hoping one will help me figure out the other.
tantek joined the channel
#
GWG
snarfed, I know it is a problem. I think I may have created it.
#
GWG
pfefferle committed a bunch of changes.
#
GWG
I wrote them.
#
GWG
But singpolyma was sending webmentions since then, so it seems to work.
#
GWG
So I can't narrow it down.
#
GWG
I decoupled the sending from the receiving code
#
GWG
I want to continue working with the existing sending code.
#
GWG
So I intend to fix It
#
snarfed
glad to hear it!
#
GWG
snarfed, they have no relationship.
#
GWG
But the current code lacks error handling and unit testing beyond the endpoint discovery..
#
GWG
Wait
#
GWG
snarfed, can you test sending with webmention.rocks?
#
snarfed
GWG: sure. i expect it won't send though, so i doubt that will help
AngeloGladding joined the channel
#
GWG
Well, will be dissecting the code this weekend and will send anything I learn to the issue.
#
snarfed
great!
#
GWG
snarfed, I also haven't forgotten about Micropub
#
GWG
But it is WordPress 4.7 season
#
GWG
And after I got jorbin to IWC, I thought I might be able to work on my agenda
#
snarfed
eh it's always season for some release
#
GWG
snarfed, agreed. But the window to talk feature projects is smaller.
#
GWG
I have something on track for 4.7 that would let me delegate Pingbacks to webmention.io
#
GWG
A lot of the tickets I am gardening are webmention related...if subtly.
#
snarfed
GWG: confirmed, doesn't send to wm.rocks either. https://snarfed.org/2016-04-11_webmention-rocks
#
Loqi
[Ryan Barrett] Webmention Rocks!
#
snarfed
it did back in april when i original posted that. not sure which plugin version i was on then
#
GWG
snarfed, will get back to you. I know what changed since April.
#
snarfed
thanks! no guarantees, i may have been running code from before april
#
GWG
snarfed, what do you think about logging sent Webmentions?
#
snarfed
i mostly just care about actually sending them :P
#
GWG
snarfed, to know what is going on, I need to have better debugging that can be turned on.
#
GWG
I have to write it.
#
snarfed
go for it!
#
GWG
And WordPress changed the HTTP API in 4.6 and for all I know that is an issue.
#
GWG
I doubt it, but in the June rewrite I made some changes there.
#
GWG
The committed changes we made were in June, so if there has been a problem since April...
#
GWG
WordPress 4.5 was released April 12th. Possible relationship?
#
snarfed
possible!
#
GWG
I guess I should read through 372 bugs fixed
KevinMarks joined the channel
#
gRegorLove
reads scrollback
#
gRegorLove
My ProcessWire Webmention plugin follows redirects on target_url
#
KevinMarks
I think mention tech does too
#
gRegorLove
Though that's after verification, and verification checks if the hostname matches, so it would always fail if target=t.co/foo
#
KevinMarks
Hm, an outbound redirection might be a good idea for link rot
#
aaronpk
Not really. Might as well just change the URL in your post once you detect link rot
#
aaronpk
Tho an outbound redirect would let you capture that functionality in an external service
#
tantek
or on your own page, because you could detect (like Google does) if someone clicks on a link on your page, and then quickly goes *back* to your page
#
tantek
that's a good red flag to use as an indicator that the link they clicked on is broken
#
tantek
(you don't need every user to do that - i.e. folks that just open tabs - as long as even *a few* users do the click / go back, you can use it)