#dev 2019-05-01

2019-05-01 UTC
#
@pypi_updates2 webmention-tools 0.4.1 https://pypi.org/project/webmention-tools/0.4.1/ (twitter.com/_/status/1123380766900203527)
[tantek], KartikPrabhu, [markmhendrickso, tbbrown, gRegorLove, neoaboriginal1, [kevinmarks786], ichoquo0Aigh9ie, kvazarig, leg, NBT and djmoch joined the channel
#
GWG Wonder if I can use expiring tokens as a IWC project
#
GWG Trying to think of what I can do in a day
#
GWG I have a json file of location history from Google I would like to try to get into Compass
#
GWG How do you parse a giant json file without a memory error?
#
sknebel cut it into small pieces (e.g. using jq) and load them one after another, or use a streaming parser
#
sknebel (I think jq can process files it can't load completely)
#
GWG Which would you suggest? I have not tried to do a 600mb data file before
#
GWG I then have to convert it to the format compass uses
#
petermolnar can you not get it in gpx?
#
GWG They offer json or kml
#
sknebel kind of depends. if its a format you can easily cut up, that might be easier than using a different parser and it's output. https://github.com/salsify/jsonstreamingparser would be one PHP streaming parser otherwise
#
Loqi [salsify] jsonstreamingparser: A JSON streaming parser implementation in PHP.
#
GWG The json has activity data
#
petermolnar kml is better imo; that's a known location data format and there are tools to split it I believe
#
petermolnar how are you trying to parse it?
#
petermolnar the json
#
GWG petermolnar, what do you mean how?
#
petermolnar language, code, etc.
#
GWG I am fine with PHP
#
GWG Although considering how much I rely on WordPress functions, I am probably just as well using any language
#
GWG I have not started
#
GWG I just started thinking about this
#
GWG Also, since Compass and its DB are in php, might as well stick to it
#
GWG No
#
petermolnar in case of PHP, if I remember correctly, most copies are deep copies, therefore every time you create an extra element you'll create a copy, unless you just use the reference and work on the original array. That said, 600MB is not that bad
#
petermolnar turn off Chrome, run the parsed in command line, 8GB memory should be enough :)
#
petermolnar s/parsed/parser
#
GWG I figured I might have to run locally
#
GWG I only have to run this once
#
GWG But it may set the stage for future own my data attempts
#
petermolnar btw I moved from backitude to ulogger: https://github.com/bfabiszewski/ulogger-server for the server side, https://github.com/bfabiszewski/ulogger-android for android
#
Loqi [bfabiszewski] ulogger-server: μlogger • web viewer for tracks uploaded with μlogger mobile client
#
GWG Okay
#
GWG I have come to like compass even if I want to make some changes to its usage
#
GWG Wonder if that is a IWC project
#
GWG On the other hand, looking for something that I can do in a day and demo
#
sknebel What us μlogger?
#
sknebel What is μlogger?
#
Loqi It looks like we don't have a page for "μlogger" yet. Would you like to create it? (Or just say "μlogger is ____", a sentence describing the term)
#
sknebel μlogger is client-server location tracking software for Android.
#
Loqi ok
#
sknebel μlogger << https://github.com/bfabiszewski/ulogger-android
#
Loqi ok, I added "https://github.com/bfabiszewski/ulogger-android" to a brand new "See Also" section of /%CE%BClogger https://indieweb.org/wiki/index.php?diff=60208&oldid=60207
#
Loqi [bfabiszewski] ulogger-android: μlogger • android application for real-time collection and publishing of geolocation data
#
sknebel μlogger << https://github.com/bfabiszewski/ulogger-server
#
Loqi [bfabiszewski] ulogger-server: μlogger • web viewer for tracks uploaded with μlogger mobile client
#
Loqi ok, I added "https://github.com/bfabiszewski/ulogger-server" to the "See Also" section of /%CE%BClogger https://indieweb.org/wiki/index.php?diff=60209&oldid=60208
#
aaronpk GWG: what kind of changes?
DenSchub, dougbeal|mb1, snarfed, [tantek], [Rose], [aaronpk], [schmarty], [eddie], tbbrown and [tmiller] joined the channel
#
GWG aaronpk, to my usage of it, not necessarily the code
#
GWG Although how to get a 600mb export from Google Location into it might be something someone other than I might want to know if I figure it out
#
aaronpk You can write to the file on disk any way you want, such as using the quartzdb library, or just make the file format match, it's a simple format
#
GWG aaronpk, converting a 600mb json file into geojson format separated by day is a challenge
#
GWG I need to parse each entry and write to desk
#
GWG That's why I should write the code and share it for others
#
aaronpk Yes sounds like a relatively small script
#
GWG It is the out of memory errors I worry about
#
GWG I have never tried to parse anything that big
#
Zegnat As long as you do not read the entire file at once, but use file pointers, you shouldn’t notice memory errors
#
aaronpk If you're just doing it this one time just change your max memory size :-)
#
aaronpk otherwise look at streaming JSON parsers
#
aaronpk but this is a good example of why I architected my file format as single lines of JSON, so that you never are in a situation where you have to parse a 600mb JSON string
gRegorLove joined the channel
#
GWG That's why I considered it a good IWC project
calumryan and [kevinmarks] joined the channel
#
[kevinmarks] there is a formalisation of json as lines - http://jsonlines.org/ -if the google file fits that model it may be easier.
#
[kevinmarks] I've been liking the python coroutines model for doing this kind of data munging - it's like pipes, but you write the individual bits in python
#
[kevinmarks] eg https://github.com/ukmetavote/ukmetavote.github.io/blob/master/ukge2eu.py
#
[kevinmarks] the util file is here https://github.com/ukmetavote/ukmetavote.github.io/blob/master/coroutines.py
calumryan_, [Rose], leg, [jgmac1106], [tantek], calumryan and tbbrown joined the channel
#
GWG Still trying to decide on a day project
#
GWG If I did expiring tokens and refresh tokens, wonder how I'd test it
treora and KartikPrabhu joined the channel