#barnabywaltersyou started from flat files and are moving towards DBs, I started storing everything in the DB and am getting progressively more and more sick of them
#aaronpkI'm still keeping the original data in flat files, probably won't even include that in the DB at all
#cweiskemy blog software has articles in plain html files, but for generating the stuff around it (index page, feeds, tag pages), I import them into a in-memory sqlite db
#cweiskebecause querying is so much easier with sql
#JihaisseI don't get why you don't want to use a DB ?
#Jihaisseand now, you want to return to flat files systems
#barnabywalterscweiske: I thought I would want to do tonnes of querying over my notes. turns out I actually want to do three, maybe four things: before/after datetime pagination, tagged filtering, deleted post filtering. maybe access control
#aaronpkJihaisse: note that I'm going to switch to flat file for the canonical data, and use a DB to indexing/querying
#cweiskebarnabywalters, and those four things are simple with sql
#aaronpkJihaisse: the point is that if the DB gets blown up, I can always re-create it from the raw data that's in files
#barnabywaltersJihaisse: yep, that’s the distinction — DBs good for indexing/caching, bad for long term storage of content
ryana joined the channel
#barnabywalterscweiske: getting the pagination right (matching the API I want) was surprisingly awkward with SQL, though that might be due to my lack of knowledge
#barnabywaltersand all of the above are easy (and, so far, fast) with the csv-based system I put together in an afternoon
#aaronpkbarnabywalters: have you documented that? "matching the API I want"
#barnabywaltersaaronpk: the API, or the difficulty I had making it with SQL?
#aaronpki'll try to document the others when I think of them again, Iusually open up my code and try to do something, then get frustrated at how long it will take with querying flat files and move on to something easier
#tantekaaronpk: The reasons you give re deleted posts is one of the reasons I chose one bim per file rather than one post per file. I can almost always get everything I need with one file system access to read one file.
#aaronpkyes, that makes sense. part of the problem i'm having is due to one-post-per-file
#tantekYou're using a hybrid of the *file system* (ie directory listing for the list of posts) and flat files themselves.
#aaronpkI have a lot of things that can create posts for me. my posting interface, import scripts from jawbone, etc, and I only plan to increasse the number of these things
#tantekMy home page requires only two file accesses (most recent bim and index.html home page template), nearly all post permalinks require only one.
#aaronpkI don't want to deal with having two things trying to add a post to the same file at a time
#tantekIs that possible or even probable in the indie web case?
#aaronpkin my case quite probable, imagine scrobbling songs to my server while uploading step counts live from devices on my wrist while writing a blog post
#tantekAh, that's solvable with an internal API/service on your own site to handle adding posts.
#aaronpk but I'd still have to deal with concurrency at that internal API level
#tantekWhich itself can queue them and them and have an asynchronous process add them.
#tantekThe queue you could make one file per post.
#tantekDone. Concurrency of adds still handled by the file system. :)
#aaronpk but if i make writes completly async, then I can't know the post URL in the response to the thing that made the request
#aaronpk I would have to switch to a URL scheme that is guaranteed to never generate colliding URLs (probably using microsecond-precision of the timestamp would be good enough)
#barnabywalters aaronpk: is the URL not based on the publishing date? which would be the time of the request
#tantekHence why I chose different short codes for post types. Since they come from the same source, each can predict their permalink.
#aaronpk the URL is roughly based on date. articles/notes are day-level precision with an index, auto-generated posts are second-level precision
#tantekThe post add API/service on your website would return you the computed permalink since it knows the current state of the queue.
#aaronpk omg you're right, but ugh that sounds complicated
#tantekBottlenecking writes/adds with a service/abstraction helps simplify architecture a lot and make it both more predictable and easier to debug.
#barnabywalters aaronpk: probably a stupid question, but surely writes are quick enough that you could lock the resource, making concurrent writes block until the previous one finished?
#tantekIt's only a problem if you have to scale up to tons of concurrent writes, which thankfully is not a problem for the indieweb use case.
#tantekIt's an advantage of building an indie website over a silo.
#aaronpk barnabywalters: sorta-ish... some of my writes grab data from elsewhere, like looking up my location to add geo context to posts, and those may be HTTP or at least redis calls
#tantekbarnabywalters: That too would like "just work" :)
#aaronpk also if I have to make a locking mechanism I would be super annoyed. if the filesystem blocked on the fopen for the write for me it would be better
#barnabywalters tantek: currently Taproot stores them in post YAML files, but I’m moving to a separate archive inspired by aaronpk’s one, but which stores multiple copies of resources over time
#tantekAaronpk yes that's exactly my worry and I think I'm convinced.
#aaronpk yeah, I definitely do not recommend storing in the same file as the posts
#tantekbarnabywalters: I think for now I will only store the latest version of a comment etc. from webmentions.
#barnabywalters tantek: that works. it leaves you open to people remotely deleting the content, which may or may not be a good thing
#tantekI'm going to go with being ok with that for now.
#tantekEg if you or aaronpk deleted a comment, I figure you have a good reason for doing so and am ok with dropping it.
#barnabywalters(nasty screenshot only as I’ve not got public pipe saving working yet)
josephboyle joined the channel
#tantekaside: speaking of tag aggregations (I think aaronpk mentioned it as a use-case), my goodness FB's /hashtag/ pages suck. I get empty results nearly all the time.
#tantek(usually FB doesn't launch or let something live that sucky)
#barnabywaltersdiaspora still has the only good implementation of hashtags IMO
#tantekalso, aaronpk, barnabywalters - do you know/remember when you started linking hashtags on your posts/notes to tag aggregation pages on your own site?
#tanteknow that we have a bunch of us POSSEing with permashortlinks or permashortcitations of the form (ccTLD postID) or (ccTLD/postID), what if we formalize that as community-driven Twitter Annotations reborn?
#tantekthat is, we formalize the format of (ccTLD postID) or (ccTLD/postID) at the end of tweets
#tantekand then we formalize how to get "more info" (i.e. annotations) about the tweet from that URL in the parens.
#tantekboom - did we just replace the defunct Twitter Annotations?
#tanteke.g. what if we got Twitter clients etc. to start to "read" our "annotations" and do things with them, e.g. display full content (ala Weave, but in every Twitter client).
#tantekhow hard would it be to convince 3rd party Twitter clients (e.g. Hootsuite) to adopt it?
#tantekand if enough clients (and users) adopted it, perhaps Twitter would too? (as they did @-replies and #-hashtags)
#tantekis considering switching from (ccTLD postID) or (ccTLD/postID) to see if he still gets complaints from folks about the link being clickable in tweets even when there's no more content.
#Loqitantek meant to say: is considering switching from (ccTLD postID) to (ccTLD/postID) to see if he still gets complaints from folks about the link being clickable in tweets even when there's no mtoe content.
#tantekwhat if we explicitly include the protocol in permashortlinks when there's more content? e.g. (http(s)://ccTLD/postID) - but still keep the whole thing in parens? That would cause it to auto-link in more POSSE destinations (e.g. FB).
ozten joined the channel
#tantek!tell benwerd since you most recently implemented and iterated on permashortlinks/permashortcitations, interested in your thoughts on moving them forward in a more unified fashion in this way: https://indiewebcamp.com/irc/2013-11-19#t1384879473
#mathpunkI want to roll everything myself as much as possible-- I'm interested in learning js/clojurescript, html, the whole shebang.
#barnabywaltersmathpunk: nice! so HTML is an excellent place to start
#mathpunkI hid from the internet for a few years when it was really starting to get interesting and now I want to return to alternate 2003 and build from there :D
#mathpunkStep Now: Staring blankly at emails re: my domain name transfer from godaddy to hover and not at all sure what they're asking me to do... hm. [ ]
#snarfedi use wordpress and pfefferle's plugins, i can vouch for them and answer q's
#snarfedesp if you're interested in frontend stuff like JS, i'd definitely suggest trying out SemPress and the wordpress-webmention plugin to get microformats2 and webmentions out of the box
#snarfedand then you can replace some/all yourself as you go
#snarfedyou were right about the twitter etiquette of link meaning additional content, but it looked like only a small fraction of people ever actually felt strongly about it
#snarfedeh personally i just put the full URL in. my twitter followers aren't hardcore enough about twitter or social media in general to care
#snarfedbut i'm following closely because i'm implementing the original-post-discovery alg
#snarfedre twitter annotations 2.0, the auto-expanding sounds similar to twitter cards, right?
#snarfedwhich are twitter-proprietary, granted, but orthogonal to permashortcitation, and have corollaries in other silos (e.g. FB's OpenGraph tags)
#mathpunksnarfed: I have not administered my own WP site... I just flapped my hands at a friend. Sooo... I guess I need to figure out how to point my domain at my vps, somehow without breaking its Google Apps connection.
#mathpunkNot that I'm keeping Google Apps, because INDIE, but step-by-step...
LauraJ and paulcp joined the channel
#mathpunkStep Now: Figure out wth DNS and Nameservers are and how to edit them on my Hover.com account. [ ]
#snarfedagreed! there are lots of tutorials and docs for that kind of stuff, e.g. pointing the main web site for a domain under google apps
#snarfedand you don't necessarily have to give up google apps. private applications you use, like gmail, google calendar, etc, don't exclude owning your own public-facing identity on mathpunk.net
#aaronpkuses google apps for @parecki.com stuff still, the nice thing is nobody really knows or cares, and I can switch out at any time
#snarfeddomain is the most important part. then comes putting the public (or semi-public) content you create and publish on it instead of (or in addition to) silos like FB and twitter
#snarfedmathpunk: yup, looks right. consider maybe doing it with your friend, either virtually in person, since it sounds like they helped set it all up at the beginning
#mathpunkI moved my registrar from godaddy to hover, and I am currently staring at the "edit stuff" page, afeared
#snarfedand one step at a time is always a good idea
#mathpunkoh, this ns1.dreamhost.com thing that's ON the hover.com site
#snarfedugh yeah, moving domains around btw registrars is always painful
#aaronpkthat means dreamhost is managing your DNS, so you can go there to change stuff
#mathpunkhurm. but I've never made a dreamhost account. soooo, this must mean that Eric (friend in question) set up hosting there back when I was all, "ugh i don't wanna know just make it do the thing"
#mathpunkok. I guess I'm stuck until he gets back to me about this then. soon enough I'll have everything I need in order to bork everything ;D
#aaronpki wish someone made this a lot easier to deal with and understand
#aaronpkit's not hard once you get your head around it, it's just that there are a whole bunch of pieces to learn in the first place and no good way to make sense of it quickly
#mathpunkI'm viewing the Internet as several decades of accretion of nerds arguing, and thus I have attained a certain inner peace
#mathpunkI think this is less worse than when I first decided I wanted to get into this webdev thing, and there were all these frameworks to make things "easier"
#mathpunkbut of course, they were making things "easier" for people who already knew what the problems being solved were
#mathpunkI'm going to get my site pointed correctly
#mathpunkI'm going to learn enough d3.js to do viz under an informal apprenticeship with a data scientist
#mathpunkI'm going to build a webapp on riotJS with a dev friend
#snarfedyeah. i have to admit, i'm not totally on board with the no display: none thing
#snarfedit seems pretty reasonable for occasional things like e.g. indienews syndication links
ozten joined the channel
#tantek.comedited /permashortcitation (+626) "add notes about benwerd's switching between permashortcitations and permashortlinks and then dropping them about a week later." (view diff)
#tanteksnarfed - yeah display:none-ing them seems wrong