#dev 2016-08-13

2016-08-13 UTC
KevinMarks, KevinMarks_, doesntgolf and ChrisAldrich joined the channel
#
loqi.me
created /Robert_Scoble_Effect (+240) "prompted by ChrisAldrich and dfn added by ChrisAldrich"
(view diff)
miklb, KevinMarks_ and snarfed joined the channel
#
snarfed
hey aaronpk, for indieauth's auth flow, if i don't include a state param when i redirect, but it gives me a state param value back in the callback URL, is that expected?
#
snarfed
did that maybe change recently?
gRegorLove joined the channel
#
snarfed
interesting, doesn't always happen. looks like this was an exception
#
snarfed
happy to send details if you want
KevinMarks, tantek_, tantek and cmal joined the channel
#
tantek
hello briefly #indieweb-dev!
#
voxpelli
hello tantek!
#
cmal
hello people:)
#
cmal
hmmmm, it seems surprisingly tricky to insert data in the middle of a file
#
cmal
I'm genuinely surprised, out of all the bottlenecks I was expecting… this, I was really not expecting :D
#
@franckpaul
@elpep fais jouer la garantie ! Fais-toi rembourser ou mieux, publie un billet et on fera des webmentions … #ohwait
(twitter.com/_/status/764429685526827008)
#
cmal
so if anyone has pointers (pun intended) regarding a filesystem or just about anything that could allow me to inject binary data at a certain point without overwriting the data already present, and without requiring to read/copy the entire file content… :-/
#
cmal
(full-disclosure: I was just trying to implement some binary search to find entries (works fine) and insert entries (doesn't) in huge ordered collections
#
cmal
or I could just give up on the time-ordering but then, if I'm adding an article to a collection (say, a tag) it will appear at the top of the collection (which is expected in some collections for reshare purposes, but I would not expect this behaviours in streams (as defined by ActivityPub)
#
cmal
the closest I've found so far would be FALLOC_FL_INSERT_RANGE (supported by ext4 and xfs), but the practical limitation is that the inserted data must be a multiplication of the logical block size (to keep block-alignment)
#
cmal
so that would mean inserting data in the middle of the file would require to allocate N blocks and rewrite there data from the next blocks until we reach alignment, which is probably less horrible than rewriting all the data until EOF, but is tricky when the size of data doesn't really fit in the block (for example, 200 bytes blocks inserting 199 bytes data would be terrible)
mindB joined the channel
#
cmal
or my collections could just be comprised of symlinks to smaller indexes which could then be rewritten as needed, I guess?
#
sknebel
cmal: maybe there is something tricky you can do with sparse files, at least for a while.
#
sknebel
cmal: also, look into how real databases structure their data on disk for inspiration
#
cmal
sknebel: I'll check this out, thanks :)
#
sknebel
an external index strukture doesn't have to be so bad
#
cmal
yeah I'm not really convinced by sparse files, mostly because the program running will just see a bunch of empty bytes (which somewhat mitigates my attempt to move throughout the file using fseek)
#
sknebel
yeah, you'd need to store offsets somewhere. and then you could just have random order and keep the data-offsets in an external file...
#
cmal
not sure I need them at all, I could just implement binary search on top of these chunks, first looking into the middle of the middle-chunk, etc…
#
cmal
or I could indeed store the offset along with the chunk reference in the main index and that would make finding the right timespan even easier I guess
doesntgolf joined the channel
#
cmal
yeah I don't know, I'll do some thinking, some digging around and some experiments and we'll see :D
#
cmal
thank you so much for the ideas sknebel :-)
doesntgolf, KevinMarks, ChrisAldrich and KevinMarks_ joined the channel
#
KevinMarks_
If you're doing your own block management you're recreating the file system, but without the hardware coupling. You're probably better off just rewriting the file
#
aaronpk
yeah that sounds like a path i never want to go down :)
#
aaronpk
in my QuartzDB i only ever append to the file. i made a "maintenance task" that I can run periodically to re-sort the file if I'm worried that some data was inserted out of order.
#
aaronpk
that task really just re-creates the file from scratch
#
KevinMarks_
Right, which is pretty much what databases do too
#
KevinMarks_
Defragmentation
#
KevinMarks_
Though on ssd that is moot really
#
aaronpk
well i want the file in sort order so it's easier to seek through in my code
#
aaronpk
i just do a simple iteration over each line
ChrisAldrich joined the channel
#
cmal
I'm actually running down the path of a collection comprised of different chunks (each one in a file) and I think it's not such a bad way to deal with the issue at all
#
cmal
I mean, let's talk about it again when I'm done implementing and I have actual code to show :)
#
cmal
(hopefully just a few hours/days of messing around ^^)
#
sknebel
cmal: since I don't think I mentioned it: my solution for time-ordered posts is a directory full of symlinks, named after the timestamp ;)
#
cmal
I've thought about this, but then depending on your filesystem your folder may end up full
#
cmal
also I'm curious, how do you retrieve the latest articles?
#
aaronpk
the main limitation i've found with working with filesystems is having a single folder filled with tons of files
#
aaronpk
so i always break mine up, usually by year/month/day
#
sknebel
cmal: right now I don't have that many posts, once that becomes an issue I'll probably add directory layers like aaronpk said. Or replace the directory with sqlite, or ...
#
cmal
fair enough :-)
#
sknebel
I can always get the timestamp from the post metadata, this is just for easy lookup
#
sknebel
(and yes, listing all files in a directory is a relatively slow operation, but I'm sure there are a lot of relatively slow operations in my setup ;))
doesntgolf, KevinMarks, ChrisAldrich and cmal joined the channel
#
aaronpk
wow neat SVG tricks, i bet KevinMarks would like this http://codepen.io/tylersticka/pen/NAojkB?editors=1100
#
Loqi
[Aaron Parecki] Centered and Cropped Thumbnails with CSS https://aaronparecki.com/2016/08/13/4/cropped-thumbnails.jpg
#
KevinMarks
Can't you just use object-fit:cover?
#
aaronpk
this has better browser support
#
cmal
aaronpk: there's also https://github.com/imazen/imageflow, but I'm not sure whether the project has stopped or is just on summer vacation :)
#
cmal
it's not much at the moment, but the goals and means expressed are very interesting
#
aaronpk
that's overkill :)
#
aaronpk
and holy crap complicated
#
cmal
depends for what precisely
#
cmal
yeah okay that's a fair point :D
#
aaronpk
also doesn't mention cropping to squares
#
aaronpk
for all their talk about efficiency, their website is super slow https://www.imageflow.io/
#
cmal
that's just a jekyll site, doesn't reflect the code :P
#
aaronpk
uh huh
#
aaronpk
it does look like a neat project though
#
cmal
agreed
#
cmal
as I said, it's more the goals and means that are interesting more than the implementation itself (at least for the moment, we'll see what it turns out to be a year from now)
#
aaronpk
"face-aware cropping" cool
#
voxpelli
Face-aware seems fairly standard nowadays :)
#
aaronpk
well i'll keep an eye on that and come back to it in a year or so when i finally get around to starting my shoebox project
#
voxpelli
Been deploying face-aware cropping on sites for the last 3-4 years I think, including on my current newspaper job
#
aaronpk
nice, what tools?
#
voxpelli
Tricky thing with image flow is their Affero license – I won't go near anything with that
#
voxpelli
aaronpk: Cloudinary at first, now Imgix, both hosted solutions
#
voxpelli
But there are face-detection scripts one can employ oneself
#
aaronpk
i'm pretty pleased with my flickr-like autolayout that avoids needing to crop images at all
#
aaronpk
i'm almost ready to launch it on my site actually
#
miklb
I looked at Imigix as they have a Jekyll plugin
#
aaronpk
imgix is kind of expensive. not sure i would use that for a personal site
#
voxpelli
Both Cloudinary and Imgix API:s are the URL one loads the image through so easy to use anywhere
#
miklb
that was my final thought, though maybe because I've never served any images, they haven't asked for money after the trial :)
#
cmal
voxpelli: what's wrong with the AGPL?
#
voxpelli
I used Cloudinary on personal stuff mainly because they let me upload and store at their place as well
#
voxpelli
Which I needed due to hosting stuff at Heroku
#
miklb
voxpelli did you move your blog from gh?
#
voxpelli
cmal: the way it can taint code and force one to release it wherever one uses it
#
voxpelli
miklb: no, meant personal as in not my day job, eg. the freelance work I did at the site for The Conference
#
voxpelli
My day job uses Imgix
#
cmal
and what's the problem with releasing code? (I'm sorry, unless it's a pro-propriety argument I don't see your point)
#
voxpelli
cmal: when it comes to GPL the web world has had a very easy ride as the server-client pattern meant they never "distributed" any code
#
voxpelli
So only with AGPL do one get the true feeling of GPL
#
voxpelli
And I'm no big fan of forceful freedom, I'm more a fan of freedom to be free
#
cmal
still, I don't see the problem with releasing server code (that's even a common practice nowadays)
#
cmal
but I get the political argument
#
miklb
don't see many new projects adopt GPL for those reasons IMO
#
aaronpk
hey this looks pretty neat https://github.com/thoas/picfit
#
voxpelli
risk assessments etc makes it a lot harder to include AGPL code than to include MIT/BSD one. I barely know where to start to get an approval for that kind of code inclusion
#
voxpelli
I much rather have everyone be able to use my code and to contribute to it, no matter if their legal or business teams find the risk of AGPL to be acceptable or not
#
cmal
although, I'd say a freedom-enforcing mechanism like copyleft, which only practices constraints on those trying to privatize stuff, is actually the closest you get to "freedom to be free" in the context of systemic oppressions (like standard copyright when it comes to code)
#
voxpelli
aaronpk: Nice, I think Will Norris has one as well: https://willnorris.com/go/imageproxy
#
Loqi
[Will Norris] imageproxy is a caching image proxy server written in go. It features: basic image adjustments like resizing, cropping, and rotation access control using host whitelists or request signing (HMAC-SHA256) support for jpeg, png, and gif image f...
#
voxpelli
cmal: the classic case of MIT/BSD vs GPL :) And funnily enough those using AGPL the most is those who wants to keep their open stuff as private as possible ;)
#
cmal
yeah, long story
#
voxpelli
I've only seen AGPL used when one also sells a proprietary license for the same software as well, like Neo4j and this Imageflow hing
#
miklb
wait until you throw Apache Software License into the mix ;-)
#
cmal
anyway I'll settle for saying this is just legalese bullshit anyway and once we've abolished the State we won't have to talk about such boring lawyering anymore, deal?
#
voxpelli
miklb: that one actually does some good stuff ;) It handles patents
#
miklb
yes. I don't disagree with its use. Just adds another layer to the discussion.
#
voxpelli
Indeed it does :P Happy that software patents don't exist here in Sweden
#
miklb
We chose ASL for Habari
ben_thatmustbeme joined the channel
#
miklb
partly because we thought about getting into the Apache Incubator, but mostly for the freedoms it affords
#
voxpelli
I think Google picks ASL and I guess that can have its benefits over eg Facebook that picked a standard BSD for React, but with a custom patents clause that has caused some drama
#
miklb
truthfully my eyes glaze over license discussions after a bit :-)
#
voxpelli
Yeah :P
#
voxpelli
At the end of the day I just want to code and not get sued for doing so :)
#
cmal
as look as share your code under any kind of free license and build blocks to tear down GAFAMs I think everyone will be okay with it ;)
#
voxpelli
aaronpk: I think Imgix uses GPU:s as well, at least I hope so considering their servers: http://photos.imgix.com/racking-mac-pros ;)
#
aaronpk
holy cow
#
sknebel
that Imgix stuff always confused me (MacOS APIs can't be *that* superior to make that worth it, can they?), but that's the kind of thing GPUs are really good at, so if you have to do it a lot...
#
aaronpk
okay that was a fun rabbit hole
#
aaronpk
now back to what i was actually doing...
ChrisAldrich joined the channel
#
aaronpk
aaand got it
#
aaronpk
i really wish css pseudo elements could contain html
#
aaronpk
although if that were possible i may have just coded an infinite loop in css
#
aaronpk
omg i think i'm going to be able to launch photo albums
doesntgolf and ChrisAldrich joined the channel