#dev 2019-04-04

2019-04-04 UTC
oodani joined the channel
#
GWG
aaronpk: How do you want to store your weather data? I see your notation about adding it to compass as an open issue
snarfed, dougbeal|imac, [tantek], [kenbauer], tw2113, jjuran, danyao, ingoogni, cweiske, [Rose], [xavierroy] and KartikPrabhu joined the channel; mblaney1 left the channel
#
@johanbove
↩️ Done! Implemented the Microformats to my Jekyll blog posts and also added http://webmentions.io links. Excited to have joined the #Indieweb!
(twitter.com/_/status/1113704670701264896)
#
aaronpk
GWG: in my posts it's stored as another property along with everything else. I also have a separate Compass database that stores the weather info every five minutes or so, I think it stores the JSON that Atlas returns
swentel joined the channel
#
swentel
morning
#
Loqi
good morning!
swentel, [kevinmarks], [Rose], barpthewire, swentie and BenLubar joined the channel
#
GWG
Morning swentel
[jgmac1106] and swentel joined the channel
#
GWG
Continuing my two projects at once work
#
GWG
Does the EU have a public/government weather service with a free API?
[kevinmarks] joined the channel
#
cweiske
that's soon not EU
#
[kevinmarks]
working on that
#
GWG
I was able to hook in the US National Weather Service
#
GWG
Want to find something that might not have rate limits or insane costs for Europe
[Rose] joined the channel
#
[Rose]
I find Dark Sky to be pretty accurate for me
#
[Rose]
The Austrian government has a whole website of free APIs you can use, but it seems weather doesn't make an appearance except as a historical "statistical book"
#
GWG
API limits
#
[Rose]
Even if each user provides their own API key? That's how you have it set up now
#
GWG
I have Dark Sky, openweathermap, APIXU and the US National Weather Service
#
GWG
[Rose]: I am maxing out API keys on my own
#
Zegnat
I feel like some governments here may have free APIs. I seem to vaguely recall swedish met office (SMHI) had something too
#
GWG
Let's assume that I am tracking my location in 5 minute intervals like aaronpk
#
GWG
How many API calls is that a day?
#
[Rose]
I would personally limit the weather calls to every 30 minutes, which is 48 calls
#
GWG
I am thinking of embedding them in with the location
#
GWG
So it is weather where I am
#
GWG
I like that feature of aaronpk's setup
#
[Rose]
Though that does take you over the 1000/month Dark Sky allows, but every hour is still quite accurate, and if you make a larger call you could use it to backfill other data
#
[Rose]
That is a nice feature
#
GWG
Hmm...it is 1000 per day or per month?
#
[Rose]
Per day
#
[Rose]
I think every 5 minutes is ok then, every 5 seconds might cause problems
#
Zegnat
Oh, wow, SMHI is putting all their data under CC0 apparently. I am going to look into this. https://opendata.smhi.se/apidocs/
#
GWG
Gpslogger for Android can be programmed not to send data if stationary
#
Zegnat
SMHI was also nice enough to make all their API docs English
#
GWG
Nice
#
GWG
I am taking notes
#
GWG
But the only thing I need is to add a pull to Compass that gets the data
#
[Rose]
Looks like the ZAMG also made their weather available through the API, documentation is... scarce though
#
GWG
I am next heading to Berlin
#
GWG
How are the Germans doing with weather APIs?
[grantcodes] joined the channel
#
[grantcodes]
!tell doubleloop You are exactly right about if I build granary integration into together then it is only in together. An ideal way would probably be to have an external client that could work on it's own. But perhaps with a simple rest api or something that other clients can integrate to make life easier for their users?
#
Loqi
Ok, I'll tell them that when I see them next
#
Zegnat
Wow, German DWD also takes open data seriously https://opendata.dwd.de/weather/tree.html
#
Zegnat
Haven’t seen a straight forward API though
#
GWG
Oh well
#
Zegnat
Apparently there is an API, TV channel WDR ended up linking to the documentation: https://github.com/wdr-data/unwetter/wiki/DWD-API
#
[Rose]
Aha, figured out the API for ZAMG, you get a CSV file and the contents change every hour. Why they don't use JSON like everyone else I have no idea, but it's a start!
#
Zegnat
This sounds like we need an etherpad or wiki page or something to collect all APIs
#
GWG
Madness
#
GWG
Zegnat, I would love one for weather and location
#
[Rose]
I could build a "translator" for it, but they also only provide data for the capital of each state
#
GWG
I have a list of ones I have used or noted
#
Zegnat
Sounds like that is a prime candidate to start documenting then, GWG!
#
[Rose]
(And a few other weather stations, I presume for skiing purposes)
#
[grantcodes]
Zegnat: There are already a bunch of lists of open apis somewhere
#
GWG
Zegnat, I just don't know where
#
Zegnat
[grantcodes]: specifically for freely accessible weather data though?
#
Zegnat
GWG: you can always create a wiki page nested under your profile page on indieweb.org
#
[grantcodes]
Not sure, I'm sure I've seen one or two that are categorized.
#
[Rose]
Point of note for GWG: This also delivers the elevation
#
[grantcodes]
Just means someone has potentially already done most of the work for you 😛
#
GWG
[Rose]: Did you know I use elevation APIs?
#
Zegnat
While true, I am having a very hard time finding this potential other person though, [grantcodes] ;)
#
[Rose]
I remembered you were interested in elevation at some point
#
Zegnat
GWG, example of nested page on my profile: https://indieweb.org/User:Vanderven.se_martijn/feed-entries-with-post-type-video - that is where I dumped my thoughts and journey when trying to create YouTube->jf2
#
GWG
Altitude, but elevation was a good estimate
#
[Rose]
Darn it, they use elevation here to mean altitude. False friends in foreign languages confuse me
#
[grantcodes]
https://github.com/toddmotto/public-apis#weather doesn't include local authority type apis by the looks of it but a start
#
Loqi
[toddmotto] public-apis: A collective list of free APIs for use in software and web development.
[davidmead] joined the channel
#
GWG
I think we can benefit from our own list
eli_oat joined the channel
#
[kevinmarks]
This sounds like worth building a microformats wiki page for weather formats per the process there - gather existing sources and formats
[schmarty] and tw2113 joined the channel
#
GWG
[kevinmarks]++
#
Loqi
[kevinmarks] has 15 karma in this channel over the last year (37 in all channels)
#
GWG
I don't think I have permission to do that though
#
Zegnat
If you do not already have an account with all the permissions, anything mf wiki is going to be hard, yes
willnorris joined the channel
#
GWG
IW wiki for now m
#
Zegnat
Sure. We probably have a weather page you can add to. Or add -developer to the wiki page name and start a new page just for development purposes (like /Webmention has)
[jgmac1106] and [kimberlyhirsh] joined the channel
#
[jgmac1106]
adafruit micrpub weather rock...not publishing weather on my blog until i build that
[kenbauer], ingoogni and [eddie] joined the channel
#
[eddie]
Hmm, man. I'm trying to figure out how to get a free standing serverless Media Endpoint and it is not going well.
#
[eddie]
It seems like all the "serverless" providers max out at 10mb per payload (which in practice ends up about 5mb of actual file size). which is not very big for a media endpoint
#
[eddie]
The other option was to try to upload directly to say Amazon s3, so I thought my Micropub endpoint could create presigned s3 upload urls whenever apps do the config query
#
[eddie]
But those all expire after 7 days
#
[eddie]
and apps don't really expect the media endpoint urls to expire
#
Zegnat
Oh, interesting issue. Wonder what the answer is to: what if the media enpoint URL changes?
#
[eddie]
I think the current expectation is that people log out and back in
#
Zegnat
I mean, there is a reason why it is defined seperately from the micropub endpoint. Specifically because you may want to use separate storage and so on
#
Zegnat
So it makes sense for clients to catch that somehow
#
[eddie]
and like Quill has a "reset" button I believe to requery the config
#
[Rose]
I dislike having to log out and in again - I would prefer to be able to click a button (hidden within settings) to look for the changes
#
[Rose]
Or perhaps clients should automatically check to see if all is well from time to time, which if the timeout could be set in q config might alleviate Eddie's issue
#
[eddie]
ohhh the reset login on Quill still logs you out
#
[eddie]
That would be interesting
#
[Rose]
It would, not sure how useful it would be
#
[Rose]
Perhaps instead a "lastUpdated" field
#
[Rose]
If updated since the client last checked, check again, but then the clients need to decide how often to check for updates to the config
#
[Rose]
Which leads us back to the timeout type option
#
[eddie]
maybe like you said, that's the option. The Micropub config query tells the client how long the current config options are valid for
#
[eddie]
when it's "expired", the client should refetch the config details
#
Zegnat
They have the same issue with access tokens that time-out, btw
#
[Rose]
And perhaps there should be a "standard" value for if it's not set?
#
[eddie]
That's true, and we don't currently have a solution for that outside of OAuth's refresh tokens. But we haven't even thought about how those should work
#
Zegnat
You could write an entire post, hit submit, only for the client to find your token has expired. I wonder how that is handled today. It would suck for you to be suddenly logged out and for the post to be lost
#
[eddie]
I think in most cases that's exactly what would happen
#
Zegnat
The solution is to ask the user to reauth, when you lack refresh tokens. The important part is just how you handle the request to reauth
#
Zegnat
Now I want to make a micropub endpoint specifically for testing this.
#
[eddie]
I guess in Indigenous for iOS it would present an error which would allow you to save your post as a draft
#
[Rose]
What happens if you write a post, hit submit, and then the endpoint is invalid?
#
[eddie]
although then to get a new token you log out and back in and I'm not sure what happens to the draft when logging out
#
[eddie]
probably it's gone because I think it's tied to the login
#
[eddie]
which is definitely an issue
#
[eddie]
especially if I end up updating drafts to be support by the server because then you can't save a draft without a server
#
Zegnat
Eh, that sounds like a bad feature. I often write draft emails on the train, where I would not have access to a server either because of lack of connectivity
#
[eddie]
That's true, I guess drafts should just be "synced" to the server
#
[eddie]
so you can have offline drafts but then when there is a connection, if enabled, it would sync the draft content into posts on the server
#
[Rose]
Personally I would try for a workflow of:
#
[Rose]
• Client checks if current tokens/endpoints are valid
#
[Rose]
• If not: save draft locally somehow, prompt user to re-auth
#
[Rose]
• User starts writing post
#
[Rose]
• If yes: save draft on server if supported, locally if not
#
[eddie]
Yeah, that's a good workflow
KartikPrabhu joined the channel
#
[Rose]
And for something like Quill, if it finds the endpoints are invalid I would hope we could extend it to give the user a .txt file download they can save
#
[Rose]
Maybe not the most elegant of solutions, but it means they can easily copy/paste the data back in after re-auth if the draft can't be maintained any other way
#
Zegnat
Not sure how it would check the endpoint... I guess it does an info query request and if that fails assumes the entire mp endpoint to be down?
#
[eddie]
Yeah, I would assume the config query will always represent the status of the Micropub endpoint
#
Zegnat
so you’d have to do the config query with the access token included, so it can tell you whether it will accept that token, I guess?
#
Zegnat
As it seems unwise to me to do a whole token endpoint discovery system for verifying that
#
[eddie]
right now all config queries require the access token
#
[eddie]
because it can contain private information
#
Zegnat
Ah, gotcha, I wasn’t sure it always required that
#
Zegnat
Was about to look it up
#
[eddie]
ohhh yeah I guess I'm not sure if it has "always" required that but it has in recent history
snarfed joined the channel
#
Zegnat
I think my micropub receiver doesn’t even support config query at all, that is how old of an implementation it is
#
[eddie]
for OAuth 2.0, when providing an access_token that expires, it will return an "expires_in":3600 attribute. So if we started paying more attention to that in Micropub clients, we could also add that same thing to the Config query
#
Zegnat
That would be a good indicator of an auto-expiring token, yes
#
[eddie]
The other thing is if a token is revoked or auto-expired, it's not 100% clear but I think it looks like it should return a 401
#
Zegnat
Of course wouldn’t cover tokens that are revoked in some other way. So you will probably have to implement some generic check anyway
#
Zegnat
Oh, good question, my first thought was 403, because you are sending along authentication, it just isn’t valid for the action you want to do
#
[eddie]
I know Nintendo Switch is using something like Open ID so it's not pure OAuth 2.0, but it's similar and I know they were sending back 401's when my tokens were expired
#
Zegnat
Double checked with OAuth, 401 is correct for “expired, revoked, malformed, or invalid”
#
Zegnat
At least as far as OAuth 2.0 errors are concerned. But I see no reason why we wouldn’t just mirror that in the micropub endpoint.
#
[eddie]
ahh gotcha. okay, so if a client receives a 401, we should walk through the authentication flow again but without actually logging out the user
#
Zegnat
That is my gut feeling. As in: you should try and obtain a new token. If that fails, that is when you log them out.
KartikPrabhu joined the channel
#
[eddie]
Yep, that makes sense
#
[Rose]
That sounds logical to me too.
#
Zegnat
And of course “log them out” may just mean you disable the post button and remove the token from storage. The client doesn’t need to throw up a boot screen and lock away drafts
#
[eddie]
Alright tried to sum up these thoughts in a extension issue (https://github.com/indieweb/micropub-extensions/issues/17)
#
Loqi
[EdwardHinkle] #17 Dealing with Expiring Tokens
#
[eddie]
I'm also gonna make one for changing config values
#
Zegnat
Amazing [eddie]++ !
#
Zegnat
[eddie]++
#
Loqi
[eddie] has 38 karma in this channel over the last year (94 in all channels)
snarfed joined the channel
#
aaronpk
Looks like you all talked through that :-)
#
aaronpk
I think more smarts in the clients about saving offline stuff is good
#
aaronpk
never assume the user has an internet connection, and also don't assume that just because a token expired that it means you should completely reset the app state
#
[eddie]
Yeah I think one challenge is because most stuff started out to be Micropub web apps without much state it didn't matter if you logged out and back in
#
[eddie]
but now Micropub apps are getting more complex, being introduced as native apps and being integrated into Reader experiences
#
[eddie]
so I think that's a big thing that starts making us rethink what used to not really be as big of an issue 🙂
#
[eddie]
because when you're building a web app you kind of assume the person has internet lol
ingoogni joined the channel
#
aaronpk
I'm just excited I can write offline apps without having to deal with the native development platforms now :-) no offense to native app development, it's just overkill in some cases
#
[eddie]
No that's definitely huge. iOS was really getting behind on offline web stuff in safari, like sadly so
#
[eddie]
There's something nice about being able to just load up a website. I just find a lot of "web app" experiences provide the functionality but they feel clunky. That is the main thing that pushes me to native apps
[schmarty] joined the channel
#
[eddie]
It should be can't
#
[schmarty]
bringing back the issue of pre-signed S3 form URLs - that sounds like plumbing that the serverless endpoint should hide
#
[schmarty]
rather than pushing complexity into the client
#
[eddie]
but can't*
#
[eddie]
So one thing I thought was having the media endpoint url be a serverless url
#
aaronpk
I have two that I use regularly now, Teacup and Airplane Time. They're not perfect by any means, but they are pretty simple and would be overkill as native apps, and also I probably wouldn't have been able to make a very good native app experience anyway
#
aaronpk
Making good apps is hard, whether it's native or web ;-)
#
[schmarty]
aaronpk: i am starting to like using shortcuts for micropub and inching closer to a good reusable do-all-the-micropubs shortcut
#
[eddie]
from what I've been reading (i haven't tested it yet) if I write a function in serverless that redirects to another url, but then I sent an upload of 10mb to it, it would get a 500 error because of the size rather than the 302 redirect
#
aaronpk
thatd be awesome. I got out of the habit of using it when they broke images in shortcuts so I'm having to train myself to use it again
#
[schmarty]
eddie: yeah, HTTP POST and redirects don't really do what you might hope anyway.
#
[eddie]
oh that's too bad 😞
#
[eddie]
well that's another issue then lol
#
[schmarty]
but in this case i would argue that this is a problem with serverless setups
#
[schmarty]
like: file the bug against AWS
#
[eddie]
haha yeah
#
[eddie]
I was hoping it was just AWS
#
[eddie]
and then when I looked around I realized all the serverless do it
#
KartikPrabhu
what is serverless
#
Loqi
It looks like we don't have a page for "serverless" yet. Would you like to create it? (Or just say "serverless is ____", a sentence describing the term)
#
[eddie]
they expect you to upload directly to their object storage
#
[schmarty]
because it is ridiculous to expect all clients to handle this odd case
#
[eddie]
if it's bigger then 5mb
#
[eddie]
I think in serverless they expect that you probably own the app
#
[eddie]
so if I were building Instaawesomegram, it would be super easy to push all uploads directly to s3
#
[schmarty]
aaronpk: last week i got a shortcut working that lets me make a read post out of any goodreads.com book link from safari by sharing it to the shortcut. iti s raddd.
#
[eddie]
the challenge is that in the IndieWeb we actually believe in decentralizing 😆
#
[eddie]
how dare us
#
[schmarty]
eddie: lol
#
[eddie]
[schmarty] That's cool!
#
[schmarty]
shortcuts' action to "run javascript on webpage" is really powerful for that kind of thing.
gRegorLove joined the channel
#
[schmarty]
eddie: that is a really frustrating set of limitations for "serverless" architectures, but i feel like it still works for many use cases.
#
[schmarty]
there are still lots of useful media that are < 5mb :}
#
[eddie]
Yeah, that's true. But unfortunately I did look at my media endpoint and find that I have a handful of photos that are > 5mb on my endpoint
#
[eddie]
So if I switched over to that my media endpoint would randomly break lol
#
[schmarty]
haha well it isn't random if you know why
#
[eddie]
I guess random as in every 10 times
#
[eddie]
inconsistently break 🙂
#
[schmarty]
i recently had a similar problem with my image proxy setup on my site
#
[schmarty]
my media endpoint takes files of any size, and i recently uploaded a > 10mb jpeg
barpthewire joined the channel
#
[schmarty]
only to find the image broken in my post
#
[eddie]
oh no!
#
[schmarty]
because the image proxy won't handle files > 10mb
#
[eddie]
ohhh that makes sense
#
[eddie]
it's tough, this might mean I just can't use serverless in this way but with my outage on DreamHost's compute platform makes me want to get as much off of my single box as possible
#
[schmarty]
the image was just fine but cloudinary was like "no way"
#
[eddie]
that's strange lol
#
[schmarty]
took me a minute to think of trying to read the headers returned in the proxied request, but they spelled out the error.
#
[eddie]
oh that's nice though, rather than some abstract error lol
#
[schmarty]
haha yep. i was able to crop the image, send it to my media endpoint via shortcuts, and edit the post to use the new image (had to do that part by hand...)
#
[schmarty]
i really appreciate your research on trying to get a media endpoint going on AWS lambda. i wonder if that should get captured somewhere.
#
[eddie]
Yeah that's probably a good idea
#
[schmarty]
i also wonder if you could stand up a free proxy elsewhere. like, glitch gives you more control and may have different limits.
#
[eddie]
save someone else that time
#
[schmarty]
or heroku
#
[eddie]
ohhh that's true
#
[schmarty]
the nice thing about decentralizing all these bits is that you can really shop 'em around.
#
[schmarty]
(even if "serverless" providers all seem to have conspired to commit max-upload-fixing)
#
[eddie]
Yeah, that is definitely a big bonus
#
[eddie]
being able to say put a Micropub Endpoint on one service and a Media Endpoint on a different service
#
[eddie]
using them for what they do best
[davidmead] joined the channel
#
[eddie]
[schmarty] I added my investigation into Lambda and S3 media endpoint usage to the wiki. Feel free to comment on if there seems to be any missing info
ingoogni joined the channel
jonnybarnes joined the channel
#
sknebel
[eddie]: if you're in AWS land, can't you make a fixed url through the loadbalancer layer (whatever that's called in that case, api gateway?) that sends traffic to the changing s3 url?
jonnybarnes joined the channel
#
[eddie]
hmm interesting. There is the Gateway API which has the same payload limit
#
[eddie]
there is also the Elastic Load Balancing but I think that's just for EC2 instances
#
@meyerweb
Is there a way in Firefox to add a site to the content-blocking exceptions list WITHOUT using the address bar dropdown?
(twitter.com/_/status/1113830765463318528)
#
KartikPrabhu
note the preferences stuff does not allow you to add exceptions only remove them
#
aaronpk
[eddie]: yeah most of the AWS stuff does not expect you to be using it standalone. They kind of expect you to use all their products together, like adding a load balancer in front of ec2 servers
#
sknebel
I guess for s3 it's cloudfront
snarfed joined the channel
#
GWG
What is Airplane Time?
#
Loqi
It looks like we don't have a page for "Airplane Time" yet. Would you like to create it? (Or just say "Airplane Time is ____", a sentence describing the term)
[jgmac1106], [kenbauer] and [Rose] joined the channel
#
[Rose]
[schmarty] I'm doing a lot of stuff with Shortcuts too, working on several mini micropub ones
#
[Rose]
I'm working on being able to use things offline, and then posting the offline ones when I get internet again
jonnybarnes and snarfed joined the channel
#
GWG
Who is using h-measure?
#
GWG
checks wiki
#
GWG
aaronpk, why haven't you posted a pushup in 5 years?
#
GWG
Not judging. Just looking for data
jonnybarnes joined the channel
#
GWG
The discussion was weather microformats so looking for information on measurements
KartikPrabhu joined the channel
#
Zegnat
uses h-measure
#
Zegnat
because duh, did you not check my h-card, GWG? ;)
[tantek], [jgmac1106], [davidmead] and calumryan joined the channel
#
[tantek]
Great micropub discussion!
#
[tantek]
Always autosave locally if you can!
#
[tantek]
What is offline first?
#
Loqi
It looks like we don't have a page for "offline first" yet. Would you like to create it? (Or just say "offline first is ____", a sentence describing the term)
jonnybarnes, snarfed and [eddie] joined the channel
#
[eddie]
I have a new Media Endpoint on Lambda road to research. Apparently Lambda@Edge (aka a lambda function running on amazon's CDN) can catch error statuses and do things like return a redirect
#
[eddie]
so I haven't looked into it quite enough yet, but I'm wondering if I can set up a url that is my "media endpoint" and then if I catch the 50-something error when the payload is too big and in that case return a redirect to my presigned s3 url
#
[eddie]
:crossed_fingers:
#
[eddie]
I still don't know how a file upload will treat a redirect
#
[eddie]
I've read mixed results online
#
[tantek]
offline first is a method of developing a website that once a user has visited it once on their device, it will always provide at least some content ([[posts]]) and or features (like [[creating]] posts) when they revisit even without internet access, using technologies like Service Workers and Local Storage.
#
[tantek]
Missed the window
#
[tantek]
what is offline first
#
Loqi
It looks like we don't have a page for "offline first" yet. Would you like to create it? (Or just say "offline first is ____", a sentence describing the term)
#
[tantek]
offline first is a method of developing a website that once a user has visited it once on their device, it will always provide at least some content ([[posts]]) and or features (like [[creating]] posts) when they revisit even without internet access, using technologies like Service Workers and Local Storage.
#
[tantek]
Feel free to improve and or add IndieWeb examples!
#
[tantek]
offline first << IndieWeb Example: {{adactio}} https://adactio.com/
#
Loqi
ok, I added "IndieWeb Example: {{adactio}} https://adactio.com/" to a brand new "See Also" section of /offline_first https://indieweb.org/wiki/index.php?diff=59032&oldid=59031
#
Loqi
Jeremy Keith
[kevinmarks] joined the channel
#
calumryan
The closest we have is possibly https://indieweb.org/Progressive_Web_App but not Offline first, likewise no page for Mobile first? They’re quite broad terms for methodology as well as technologies to cover
jonnybarnes and snarfed joined the channel
#
[tantek]
Pwa << offline first
#
[tantek]
offline first << pwa
[schmarty], ingoogni, KartikPrabhu, [dougbeal], jonnybarnes, calumryan, gRegorLove_, [fluffy], [tantek], [jgmac1106] and [kenbauer] joined the channel