#dev 2019-08-01

2019-08-01 UTC
[David_Bryant], [Michael_Beckwit, RockSteadyTRTL[6, [tantek], [xavierroy] and [Rose] joined the channel
#
GWG
Still analyzing the webmention spec and brainstorming and taking notes on ideas for improvement in our implementation.
cweiske, raucao, bear, [prtksxna], [tonz], [Rose] and [renem] joined the channel
#
[renem]
[snarfed] getting back to the AP follow from Mastodon. Did you „unfollow“ my profile [renem]@renem.net on Mastodon? If not. I don’t see any followers and expect the same behavior as I get from Mastodon.host that the follow request ends in Nirvana/Converts to a Approval required request.
#
krup
Has anyone found a particularly easy way to parse or translate micropub requests in python? I have been manually converting them to Dicts and then translating them to values that are easier for Jinja2 templates to ingest but I feel like there's a better way!
[jgmac1106], [grantcodes], eli_oat, [Sadik_Shahadu] and [snarfed] joined the channel
#
[snarfed]
[renem] sorry, i don't entirely follow that. here are screenshots showing the successful follow though.
#
[snarfed]
i noticed you said `@renem@renem.net` just now though. you need a .net on both sides, ie `@renem.net@renem.net`
ben_thatmust and [renem] joined the channel
#
[renem]
[snarfed] is the „success“ still visible when you reload the whole Mastodon page? This is, when I noticed that it switched to an „approval required“? Sorry to bug you with that but I’ve no idea whom else I could ask to answer my issue 🤷‍♂️
#
[renem]
[snarfed] that’s the view for me when I try the follow and hit the reload button in the browser after the follow. Also look at the follower count, it says 0 - 1 - 0 After reload.
[zach], [frank], [jgmac1106], [schmarty] and [kimberlyhirsh] joined the channel
#
[snarfed]
[renem] yes, i still see that `@renem.net@renem.net` has 1 follower when i reload. URL is https://mastodon.technology/web/accounts/229651
#
Loqi
René
#
[snarfed]
honestly i'm not sure what's going on, but mastodon has funny internal logic that's beyond normal AP that i don't fully understand and can't easily debug
#
[snarfed]
all i know is when i follow `@renem.net@renem.net` from my `@snarfed@mastodon.technology` account, bridgy fed gets the AP `Follow` request, sends an AP `Accept`, and everything works
#
[snarfed]
when you try to follow it, bridgy fed never gets the AP `Follow` request. so i still believe there's something unusual with mastodon.host specifically
#
[snarfed]
maybe ask its admin?
jackjamieson, gxt, [tantek] and [renem] joined the channel
#
[renem]
Will do [snarfed], thanks for helping!
#
[snarfed]
actually [renem] i see a change i can make to bridgy fed's undo handling that may help mastodon. let me try that first
#
[tantek]
want to make sure krup's Micropub question isn't lost: https://chat.indieweb.org/dev/2019-08-01/1564652665849000
#
Loqi
[krup] Has anyone found a particularly easy way to parse or translate micropub requests in python? I have been manually converting them to Dicts and then translating them to values that are easier for Jinja2 templates to ingest but I feel like there's a bet...
#
[tantek]
What is Jinja2
#
Loqi
jinja2 is a python-based templating language for HTML with rich macro support that adds {% %} syntax https://indieweb.org/jinja2
#
[snarfed]
that sounds like the expected technique. json.loads() for JSON requests, urllib.parse.parse_qs() for form-encoded. should be pretty straightforward
#
Zegnat
There is a Python mf2 validator that may give some ideas of working with micropub JSON? https://github.com/cleverdevil/microformats2
#
Loqi
[cleverdevil] microformats2: Validate Microformats2 JSON with JSON Schema, and perform Post Type Discovery
gxt joined the channel
#
[snarfed]
ok [renem] try again, might work better now
#
[renem]
[snarfed] tried it just now and it’s still the same. I try to reach out to an admin and ask them if there is anything different than on other Mastodon instances. But I doubt that I get an answer that satisfies me.
#
[snarfed]
agreed. sorry.
#
aaronpk
just to confirm, [renem] you're using `@renem.net@renem.net` not `@renem@renem.net` right?
#
[snarfed]
aaronpk yeah he is. i checked in (slack-only) screenshots
#
[snarfed]
[renem] fwiw i'm still not seeing _any_ AP requests from mastodon.host to bridgy fed, at all
#
aaronpk
ah gotcha. missed the screenshots
#
aaronpk
i wonder if mastodon.host blocked bridgy fed?
#
[snarfed]
possible
#
[renem]
Maybe I try to create an Mastodon account on another instance for another test.
[jgmac1106] joined the channel
#
[jgmac1106]
Weird swentel, aaronpk, I can't load my reading channels with Indigenous but I can in aperture
viaken joined the channel
#
aaronpk
oh no i hit the swarm API rate limit
[Rose] joined the channel
#
[snarfed]
aaronpk i feel your pain
#
aaronpk
wow i'm even on the upgraded "personal" account
#
aaronpk
"99,500 Regular API Calls per day"
#
[snarfed]
damn you're averaging 1qps? not bad
#
aaronpk
i have a bunch of logic in there to continue to poll every checkin looking for changes, since sometimes photos get added afterwards
#
aaronpk
so i probably need to tone down those checks
#
[snarfed]
oof "every checkin" 🤔
#
aaronpk
yeah, each checkin gets polled at these intervals (seconds) 1,15,30,60,120,300,600,1800,3600,14400
#
aaronpk
the way swarm works is even a checkin created with a photo at the same time sometimes the photo appears a few seconds later
#
aaronpk
but the webhook runs first, so it looks like a checkin with no photo at first
#
[snarfed]
ahh ok but finite times
#
aaronpk
yeah, it gives up after a day
#
[snarfed]
seems fine
#
aaronpk
if a checkin hasn't changed in a day, it probably won't change again
#
aaronpk
oh also it polls recent checkins looking for comments and coins and such
#
aaronpk
on a similar but different schedule
#
aaronpk
oof i don't know what to do here other than turn down the number of polls
#
aaronpk
which really just pushes out the problem to later
[KevinMarks] joined the channel
#
[Rose]
Could we have an option in our account to disable checking for coins and comments?
#
[Rose]
I don't care about coins being sent to my post, and comments happen so rarely that if they were lost I'd live without them quite happily.
#
aaronpk
ooh that'd help, good idea
#
[Rose]
I live to attempt helpfulness
#
aaronpk
ah there is a toggle for that already
#
aaronpk
but it defaults to "on" for everyone
#
aaronpk
oh gosh i found a stupid bug
#
aaronpk
my scheduler for polling checkins for changes and polling for comments are colliding
#
aaronpk
well not a lot i can do until my rate limits are reset now
#
aaronpk
20 hours 40 minutes until aaronpk's foursquare rate limits are reset
#
Loqi
I added a countdown scheduled for 2019-08-02 9:01am PDT (#6625)
#
aaronpk
oops double timezone conversion
#
aaronpk
!cancel 6625
#
Loqi
Okay, I cancelled it!
#
aaronpk
4 hours 40 minutes until aaronpk's foursquare rate limits are reset
#
Loqi
I added a countdown scheduled for 2019-08-01 5:01pm PDT (#6626)
gxt joined the channel
#
aaronpk
oh nvm, the flags are on two different tables 🤦
#
aaronpk
anyway, that's enough messing with that for now
#
aaronpk
ah, some people don't even have a webmention endpoint. i should figure out how to catch those and disable checking backfeed for them
#
aaronpk
ok that's a lot of new code. can't test for another 4hrs
#
[snarfed]
except you do want to rerun wm discovery periodically
#
aaronpk
eh, if someone feels like it they can always turn the toggle back on
#
aaronpk
there's a toggle switch in your dashboard to enable/disable checking for comments
#
aaronpk
so my new code turns it off if there's no webmention endpoint found 10 times
gxt joined the channel
#
[snarfed]
the lengths i've gone to in bridgy to avoid even trivial UI 😂 😭
#
aaronpk
i suppose it would be friendlier to show a banner that it was automatically disabled
#
aaronpk
ownyourswarm actually has a pretty extensive list of settings. it's grown over time
#
GWG
I should check mine
#
Loqi
I agree
#
aaronpk
if i hit the rate limits again tomorrow, i'm going to do something drastic and disable backfeed for everyone, and provide a notice in the dashboard, and you can turn it back on if you want
#
GWG
I am fine with that most of the time
#
Zegnat
Will be interesting to see how much you can spare by disabling the comment backfeed for those without wm endpoints. Do you think you would be able to get some sort of % of “people without wm endpoint”? Would be very interesting to see. I would have expected everyone who has micropub endpoints setup to already have webmentions endpoints
#
aaronpk
it's not easily queryable right now but i can check
#
[snarfed]
Zegnat indiemap can easily answer those kinds of questions!
#
[snarfed]
over twice as many wm endpoints as micropub
#
Zegnat
Just didn’t want to figure out the query for “how many have micropub endpoints but not webmention endpoins” (and figure out where to execute said query), [snarfed]
#
Zegnat
Also made more interesting by the fact that they would have the mp endpoint, no wm endpoint, and sign-up for a service that does backfeed over wm. Though of course some people may not know it does that, and the toggle was on by default.
#
[snarfed]
heh ok. i like to think the docs are pretty good, and it literally is just click to open the page and type SQL, with autofilled columns...but still, understood
#
Zegnat
Oh, the docs may be great, but not at 10 in the evening when your day starts at 4 in the morning ;)
#
Zegnat
Will be off in a second again, because tomorrow also starts at 4
#
[snarfed]
oh wow yeah go sleep
[grantcodes] and [tantek] joined the channel
#
[tantek]
snarfed, any chance of getting annual updates to IndieMap to answer questions like that?
#
[snarfed]
[tantek] my standard answer is yes!...as soon as i see any evidence that anyone actually uses it 😎
#
[tantek]
hmm, here's a non-use datapoint — I went to check it to help provide stats for State of the IndieWeb 2019 at IndieWeb Summit, saw that it was still 2017 data, moved on
#
[snarfed]
yeah, that's disappointing. sure, 2 years old isn't yesterday, but i think it's still useful and informative
#
[snarfed]
i think we often overindex on "freshness" in general, eg with open source projects
#
aaronpk
not to answer the question "what's changed since 2017" 😂
#
[snarfed]
hah yes for that one use case
#
[tantek]
I think asking an annual question of what's changed in the past year is a very reasonable use-case and is not overindexing on "freshness"
#
[snarfed]
it's a substantial amount of work to recrawl and regenerate. ~6M HTTP requests! over a home internet connection on a single laptop 😂
#
[tantek]
right, once a year should be ok for that?
#
[tantek]
of course I'd hope to see that 6M increase year over year 🙂
#
[snarfed]
sure, yes. i just mean that most use cases are _not_ "what's changed in the last year," and i think it's still useful for those
#
[snarfed]
i'd love to see enough evidence that people have used it for other use cases that i'm convinced to recrawl!
#
[snarfed]
i haven't yet though
#
[snarfed]
fwiw, the original crawl took me 1-2 _months_
#
[snarfed]
on and off over a much longer period
#
[snarfed]
eh ok maybe one month. still.
[kimberlyhirsh] joined the channel
#
[snarfed]
i guess i can rephrase another way: if anyone wants it updated badly enough, it's open source, i'd happily help them do the recrawl themselves!
[eddie], KartikPrabhu and [jgmac1106] joined the channel
#
[jgmac1106]
Snarfed as someone doing academic research I would help support a recrawl
#
[jgmac1106]
I get a small snapshot of a year, run some interesting network analysis but it be fun to update
#
[jgmac1106]
Really I dream of an opt-in mf2 search engine... Heck forget crawling.... What if an index was powered by micropub... Could that even be possible?
#
[jgmac1106]
Like i say index my articles and notes but not my listens... Every time I publish the "micropub search engine" adds my stuff to the index through micropub.. Of course i have no idea how big these files/tables cpuld get and what this would do to regex
#
[jgmac1106]
tl/dr yeah recrawl. Love data. Can find support if needed
#
[tantek]
more like websub search engine. no need to auth-in
#
[jgmac1106]
Actually I like the auth-in as kind of way to control a semi private/unlisted.. Or would need something in my header that equals "do not websub"...
#
[jgmac1106]
But that seems like a lot... Be cool to log into a search engine and toggle post types, "Everyone can have this, members of this group/ring can have this, nobody searches this"
#
[jgmac1106]
I don't know the tech but what if a search engine was more push in rather than crawl? Just seems like more user control.
[Michael_Beckwit and jeremycherfas joined the channel
#
[snarfed]
[jgmac1106] glad to hear it! i think we have enough support; what we really need is someone to actually drive it.
KartikPrabhu joined the channel
#
krup
something like that would splash big i think
[KevinMarks] joined the channel