#dev 2019-08-13
2019-08-13 UTC
# 
vika_nezrimaya oh well it seems like I don't have alternative slugs supported in my Micropub endpoint, what an oversight
# vika_nezrimaya it seems like I have outgrown redis...
# vika_nezrimaya and my software isn't even production ready
# vika_nezrimaya has anyone used CouchDB before here? Is it good?
itsmekntDiscord[ joined the channel
# [snarfed] vika_nezrimaya you may be interested in https://indieweb.org/database-antipattern
# vika_nezrimaya @snarfed: file systems seem to be my bottleneck on an RPi with an SD card
guilbo joined the channel
# vika_nezrimaya Because on a laptop with a normal HDD it's a lot faster
# vika_nezrimaya I just transferred it to a laptop and it became faster
# vika_nezrimaya Architecture, clock speed, but disk I/O is I think the most impacting one here
# vika_nezrimaya 'cause SD cards aren't built for server-grade applications
# [snarfed] you may also be interested in aaronpk's architecture, which uses filesystem for canonical storage, and also a db as a purely derived index + cache, to make querying easier and sometimes faster. https://indieweb.org/p3k#Indexing_and_Caching
# vika_nezrimaya I tried to transition to Redis to keep the dataset in memory at all times. But Redis isn't a proper document store, and with a proper document store it could be easier to query posts by properties (i.e. implement search or filtering by tags)
# vika_nezrimaya Right now I think I'll end up with a lot of auxilliary keys like category_{} = ["slug1", "slug2", "slug3"] to speed up tag feeds
# vika_nezrimaya I already have a "posts" hash with my posts in MF2
# vika_nezrimaya and a main feed list containing their order
# vika_nezrimaya MySQL? no thanks I've heard of enough problems with MySQL and now I'm scared of it 😂
# vika_nezrimaya but this could be a nice idea - to store posts in flat files but load them up in Redis as a cache
# vika_nezrimaya but then I lose on Redis' native replication...
# vika_nezrimaya could get out of sync between these whiles :3 rsyncing every second I think is a bad idea :D
# vika_nezrimaya and we're returning to my old design basically but with a cache DB
# vika_nezrimaya and posts in MF2 JSON
# vika_nezrimaya instead of Hugo markdown with front-matter in YAML
# vika_nezrimaya I hope my Micropub endpoint code is modular enough so I can just swap out functions for reading and writing posts to switch a DB :3
# vika_nezrimaya right now it puts the post into the "posts" hash, adds it to the "posts_order" (which is a fancy way of naming my "main feed" list), updates "categories" (used on ?q=category for autocompletion) and updates "slugs" key with alternative slugs for a post
# vika_nezrimaya If I were to switch it for local files, I would dump JSON into a file and... make some data structures for feed, probably also on the filesys
# vika_nezrimaya WAIT I GOT IT
# vika_nezrimaya I understand why my previous implementation was a snail
# vika_nezrimaya It read THE WHOLE DATASET from disk on EVERY SINGLE FEED QUERY
# vika_nezrimaya more than 200 files
# vika_nezrimaya right now my dataset is at 299 posts
# vika_nezrimaya 299 open() calls
# vika_nezrimaya it IS problematic :3
# vika_nezrimaya my dataset is 1.1M right now, this means it reads 1M of data from an SD card. I could continue storing JSON files, but build an index in Redis. In case of power outage the index could be easily rebuilt from the dataset in flat files since they contain A LOT of metadata
# vika_nezrimaya this basically means ditching posts key
# vika_nezrimaya only
# vika_nezrimaya I had f2fs on my last one
# vika_nezrimaya so that should've prolonged it a little bit
# vika_nezrimaya that really helped :3
# vika_nezrimaya I remember my first card dying spectacularly
# vika_nezrimaya I had to reboot the Pi by BLINDLY logging on tty1 with a keyboard
# vika_nezrimaya it was a headless setup
# vika_nezrimaya SSH wasn't working
# vika_nezrimaya and I didn't have a serial converter
# vika_nezrimaya didn't get to venues yet
# 
[KevinMarks] Google had a neat peano curve that favoured land for linearising latlong
# [KevinMarks] Cities are sparse at higher latitudes in either direction, so there may be a cheat there
# [KevinMarks] I have a project that is gently abusing the UK national grid, which mostly works until I make a mistake and get 0 latitude somehow and everything is super distorted
# [KevinMarks] The UK national grid is seductive as the country is small enough that you can treat it as flat and use metres and calculate distances in 2d with pythagoras without the errors getting big enough to matter, and angles just work.
# [KevinMarks] And the channel islands are just about in range, but if you include Gibraltar things go dodgy.
# [KevinMarks] Nice.
# vika_nezrimaya oh wait I don't think need CouchDB...
# 
[tantek] or there's this approach: https://optional.is/required/2010/12/13/hls-world-map/
# [KevinMarks] How often are you rewriting things, as opposed to appending them?
# [KevinMarks] Tantek's bim model may suit you if you're primarily working by date
# [KevinMarks] What Simon has been doing with sqlite may also be an option if your data is more read than written https://simonwillison.net/2017/Nov/13/datasette/
# [KevinMarks] Literally 3 lines of code https://github.com/banterability/blueshift/blob/master/index.js
# [KevinMarks] 2 if you don't export it
# [KevinMarks] var md5 = require("md5");
# [KevinMarks] const blueshift = cityName => `#$
{md5(cityName).substr(0, 6)}`;# [tantek] stumbles upon https://www.geonames.org/
# [tantek] kevinmarks, to answer your question about San Jose: https://web.archive.org/web/20131027070722/http://www.dopplr.com/place/cr/san-jos%C3%A9-2
# [KevinMarks] In any language with useful libraries (ie not js) it's no trouble
# [tantek] aaronpk, since you first had no idea and then figured it out, here's the list based on that roughly clustered by longitude heuristic I mentioned: https://indieweb.org/cities#IndieWebCamp_Cities_By_Region
# 
aaronpk And done https://pin13.net/city-color.php
# [tantek] kind of an amazing coincidence that /Portland 's color is red like a rose and /San_Francisco 's color is almost orange like the Golden Gate bridge though also kinda pink/fuchsia
[timothy_chamber joined the channel
# [tantek] cities << https://pin13.net/city-color.php
# 
Loqi ok, I added "https://pin13.net/city-color.php" to the "See Also" section of /cities https://indieweb.org/wiki/index.php?diff=64348&oldid=64344
# [tantek] aaronpk now bordered with colors from your city-color service: https://indieweb.org/cities#IndieWebCamp_Cities_By_Region
[Rose], Tevya, IWSlackGateway, KartikPrabhu, [tonz] and cweiske joined the channel
# aaronpk according to this, the plan is to move tumblr to wordpress as the backend! https://poststatus.com/automattic-has-purchased-tumblr/
loicm, KartikPrabhu and [tantek] joined the channel
[grantcodes] joined the channel
# 
[grantcodes] It's also something they probably couldn't do without the new editor. They can provide a custom UI for Tumblr using standard WordPress functionality
# [grantcodes] Or they could use the rest API
# [grantcodes] WordPress has always had a reader?
# [grantcodes] To .com not self hosted
IWSlackGateway and [tantek] joined the channel
[grantcodes] joined the channel
# [grantcodes] WordPress.com is still WordPress. But Tumblr will be the same I assume it won't be open source
# [grantcodes] You should say WordPress.org for what you're meaning. I think that's how it's generally separated.
# [grantcodes] Or WordPress core is the naming usually used for the actual main open source code.
[Rose], [Lewis_Cowles], jeremych_, [KevinMarks], jgmac1106, [pfefferle], [jgmac1106], [xavierroy], vika_nezrimaya, IWSlackGateway, [tantek], loicm and nloadholtes joined the channel
# [tantek] via sl007: ActivityPub Conference 2019 – Speakers / Talks announced https://redaktor.me/apconf/ (requires JS to view)
# [tantek] Some good topics worth broader discussion here too (noting here in the logs since content may disappear due to js;dr): “Keeping Unwanted Messages off the Fediverse” / Spam, scams and harassment pose a threat to all social networks, “Decentralised Hashtag Search and Subscription in Federated Social Networks”
# [tantek] More talk description details (without JS) in cwebber's blog post: https://dustycloud.org/blog/activitypub-conf-2019-speakers/
# 
jeremycherfas !tell zegnat Is there any reason not to download that CSS I was calling from the cloud and serve it myself, rather than do the integrity check?
# jeremycherfas Thanks. For context, this is a CSS made available by a CSS framework. I don't want to incorporate it into my build process (because I don't have one) and so I am working with the CDN while I develop. But I am unlikely to need all of it in production.
[jgmac1106], IWSlackGateway, valuemachine, rainmanj_, jackjamieson, zoglesby, [tantek], KartikPrabhu and [snarfed] joined the channel
# jackjamieson [snarfed] I have a quick question about granary. When I convert my twitter timeline to HTML using granary, some tweets are given a u-category consisting of the author's h-card. Why is u-category used here?
# 
KartikPrabhu what is person-tag?
# Loqi A person tag (AKA people tag) is a person mention that is also a tag on a post that refers to a specific person by URL rather than just a word or phrase, and is done as an explicit tagging action by the user, beyond just mentioning a person via hyperlink / h-card / or @-name, autocompleted or not https://indieweb.org/person-tag
# KartikPrabhu jackjamieson: maybe that ^
# jackjamieson When I parse those tweets through my Microsub server they end up with a category property that causes alltogethernow.io to error
# KartikPrabhu hmm I suppose Together should fail gracefully
# jackjamieson Thanks KartikPrabhu, that makes sense
# jackjamieson Looks like that's it. My understanding is that Together has fairly strict expectations for data structures, but I agree it would be better to fail gracefully. I'll create an issue in Together
# KartikPrabhu yeah. for consuming microformats it is better to not be very strict, or at least not throw up errors if the data structure is not recognized
# jackjamieson I should also see how Aperture handles that case, since the flow of twitter->granary->aperture->Together works as far as I can tell
[schmarty] and nilocDiscord[m] joined the channel
[grantcodes] joined the channel
# [grantcodes] Evening! So together does support categories. But I think it expects the category property to only contain strings at the moment, not objects
# [snarfed] lol bridgy has struggled with that before too, when it's hit objects in the url property. https://github.com/snarfed/bridgy/issues/511
sblinnDiscord[m] joined the channel
# [grantcodes] That's sort of the goal of jf2 though isn't it? To be more opinionated / strict on data structures
jackjamieson joined the channel
jackjamieson joined the channel
jackjamieson joined the channel
jackjamieson, loicm and [schmarty] joined the channel; truthDiscord[m] left the channel
# jackjamieson Thanks all for the additional context. I'll take care of this in Yarns then. GWG, do you reckon such cleaning should be part of Parse-This, or further downstream in Yarns?
# jackjamieson GWG, Thanks, I'll shift my issue over to Parse-This. I started writing a function to address this in Yarns so I'll adapt that into a PR
# alexmarcus[m] What is Together?
# Loqi Together is a social reader that was initially conceived at the 2017 IndieWeb Summit in Portland by Jonathan LaCour and several others during the Putting it all together session https://indieweb.org/Together
# alexmarcus[m] Thanks Loqi
# jackjamieson Sknebel, good point about expressing a person tag in jf2. To fit the current jf2 spec, one approach could be to simply take either the 'url' or 'name' property from the h-card. I'm open to suggestions if there are existing conventions etc.
# jackjamieson Also, regarding my original question about granary, the FAQ at the bottom of https://indieweb.org/person-tag states that Twitter handle links should not be person-tags
# KartikPrabhu I think twitter does support person-tags, and the @-mentions would simply count as mentions then
# KartikPrabhu yes
eli_oat joined the channel
# jackjamieson GWG, I think you're right that a reference makes sense. If my understanding is correct then a person-tag containing an h-card should be reduced to its URL, and the references property can contain the full h-card
# jackjamieson Thanks!
# jackjamieson I'm off for now but I'll try to write that PR this week
[snarfed] joined the channel