#butteryou'd have to run mbox2md or something to spit out individual messages.e it indexes on an as-seen basis, primarily symlinks used to reconstruct threads, and some per-address index-dirs
#shaners_I feel like there's an opportunity here to reframe the db or flat files thing.
#tantekshaners are you kidding? I can't believe how much attention it is getting.
#tantekwho knew people were so passionate about plumbing.
#shaners_Maybe there could be a two column comparison table (like when a company is compary its product to another company's product. [or different pay levels of a product]).
#kylewmshaners_: do you mean reframe it as something like "database vs flat file tradeoffs"?
#shaners_but my main point / concern here is that the name of the page (and the content of the page) doesn't really acknowledge the trade offs. OTHER PAGES on the wiki do (to a degree), but A: need more attention (from me and others) and 2: are other pages :(
#tantekkylewm - clearly I need to ship a high-performance structured-flat-file library to as step 2.
#gavincalso, I'd worry a bit about ballence... have you seen the responce? there is a frighting number of people that think web applications or sites HAVE to have databases
#benatkinmaybe we should call LevelDB and BoltDB phat file databases
#tantekkylewm - you yourself remarked at how surprised you were at the performance of my site based on dynamically building pages from a flat file storage.
#tantekgavinc - the response has been 99% emotional. "Is this for real?" "Words cannot describe" "wut" "stay away" etc. and thus indicates likely presence of Cargo Cult Programming.
#tanteksince people defend with emotions (rather than reasons) when they've adopted something for Cargo Cult reasons, rather than actual thoughtful rational reasons.
#benatkinseems to have some of the advantages and disadvantages of flat files
#kylewmtantek: i'm not sure why you said "you yourself remarked at the performance of my site" ... i'm not saying flat files are slow
#shaners_but mostly, I don't care enough (about this db/flatfile debate) to care, which is why i've not put any effort into the db side of the conversation on the wiki. i've always thought that it wasnt worth my/our time to even talk about.
#tantekshaners - I only found it worth documenting because I noticed a pattern of problems (around using databases for personal site content) that no one else had actually collected into an article - so I collected it.
#tantekFelt like one of those "denial" things in the industry.
#tantekand turns out, there very well may be some denial as evidenced by the overwhelming emotional-only response!
#kylewmmaybe a title that uses "risks", "problems", "concerns", "costs", etc. instead of "antipattern" then?
#benatkinso far I haven't seriously compared apple/oranges for storage
#benatkinthis isn't going to get me to talk seriously about it :)
#tantekkylewm - I only brought it up because a) you remarked at last HWC how flat files were giving you perf problems and then b) I noticed you switched (back) over to using a DB for your site content storage (edit today)
#tantekunderstanding the reasoning behind such decisions is useful
#kylewmso my rationale was: i was thinking about how not to have 9 zillion small files, while still making it easy to make small atomic edits (like adding a reply context asynchronously), and keep indexes searchable by type and tag
#shaners_benatkin: indieweb is more than just data ownership
#kylewmand started feeling like "i'm going to have to write an awful lot of code to reinvent sqlite"
#benatkinit's worth the occasional misunderstanding about it
#benatkinshaners_: yeah, I'm just not as interested in those parts as some, but it's probably just because of apathy
#kylewmso i switched back to sqlalchemy and deleted several hundred lines of code, which felt nice
#gavincDealing with databases is exactly why gavin.carothers.name back running wordpress on wordpress.com and not home rolled :( Databases are enough fun at work all day
#tantekgavinc - fascinating, it's due to your data being stuck in a database that's preventing you from hosting your own site on web hosting instead of wordpress.com?
#gavincnope, it was when I thought about setting up paggerduty for mine (and my wife's) blog when they went down when I screwed up a postgres upgrade :D
#gavincgetting it out and into wordpress import format seemed easier to keeping running my own database :\
#gavincand NOW, yeah getting back out of the database with comments etc is much more annoying then getting everything into wordpress xml. So likely just do it again via a wordpress xml export
#acegiakso I made a font of my handwriting and then made a webpage with the "handwritten" message on there with pictuers and had a song fom when were teenagers playing
#KartikPrabhuoh I see. fwiw I have been trying to keep all pieces of my site separate
#shaners_One could have an app for CRUD of a post type (or all posts), and a separate app that does just in and outbound webmentions, another one for posse sydication etc.
#bretdude! so cool, Ben and Erin got to talk about known with Douglas Rushkoff
#acegiakshaners_: does the network of plugins for wordpress that have been popping up count as monolithic?
#benatkindidn't realize indiewebcamp was an alternative to c2 :)
#shaners_acegiak: Meh. It's cool that WP has a plugin architecture to add new features / functionality. But ultimately they all get loaded into big app, at least from a URL sense of it.
#shaners_I.E., I can't drop a WP plugin into any other kind of app to use it. I have to use WP. Bc it's integrated at the app level, not the URL/HTML/HTTP level.
#acegiakit's been nice seeing how the plugin devs (myself included) are trying to make things that rely on each other as little as possible
#KartikPrabhuI don't have to care at all if the other person uses PHP/Python MySQl/Postgress/flat-file or even ASP
#shaners_one of the (many) knock on benefits of this approach is that we can use other people's software (as long as it reads or writes HTML+uf2 over HTTP). and vice versa. someone making php software could run our ruby app/s alongside theirs and interop at the HTML/HTTP level.
#KevinMarks_RDBMSes, though much more reliable than most user-written transaction processing code, are not nearly as reliable as a basic Web server pulling static files out a file system. Prepare to hire a half- or full-time database administratorÂ
#butterapparently, the indieweb scene is guilty of all sorts of ills and must be shamed
#butterit wouldn't shock me if a lot of google employees don't even like the products, but do it because it's how they can make a living. and want decentralized/p2p stuff for their own sites
#butteri mean, ive met CISCO, etc employees that hated the products
#KartikPrabhubutter: Shanley is right about the non-diversity here. But it is a problem that we are aware of and try to tackle
#butterthe sausage-party, and insane-rents of SF, combined with having to work on least-common-denominator, centralized/proprietary consumer-crap ..
#butterjeez. move to Sacto or at leas tsome part of the east-bay where yuppies hate or haven't found.. so you can relax
#petermolnarShanley 13 h "White men CANNOT BY DEFINITION lead the "independent web" movement to save us from the surveillance machines they built & profit from." Seriously? That woman is nuts.
#butterthe monoculture was definitely a turnoff, but i dont go ranting about it. i jsut found somewhere diverse and interesting. ie Brooklyn
#tommorrisI agree butter. I've been to SF/SV. I like SF but not SV so much. do prefer living in London. :)
#buttera filesystem is a database. so it's confusing reading that wiki-article
#butterit just happens to be a better databse for 99% of use-cases where you don't need extremely sophisticated querying-capability and want much easier backup/redundancy/basic-manipulation ability.. via all the existing UNIX tools, rsync/scp/nfs etc
#butterbetter yet, just use file: or https:// URIs and whatever index you want *and* files... SPARQL, Mongo, Redis...
#KartikPrabhubutter: speaking as a non-dev, I think of file system and databases as different things. technically they both store things but from a usage pov they are quite different
#KartikPrabhulooking at a MySQL database, still gives me shivers
#alanpearceMySQL is a bad example of a 'database' IMO
#butteracegiak: reddit is mostly open-source. except the automoderator/spambot engine/rules stuff, whether for cat+mouse or whatever
#butterand, it's probably way too huge to consider indie, ie how man IT experts and what size cluster do you need to launch reddit on your own hw?
#KartikPrabhualanpearce: it is the most ubiquitous. Also, non-dev folks are not in a position to make that judgement.
#tommorrisThe only thing I'd suggest we might learn from the database-antipattern twitter shitfest is that when there's a topic where people in the community differ (I use postgres, don't have an issue with it), it's probably best to, say, post the bulk of the content on our own sites and then link out. so instead of a page called "database antipattern", we just have
#tommorrisa page called databases, and then if you feel strongly that databases are a bad idea, post it on your own site and link it.
#tommorriswe've done that for a few other topics on the IWC wiki
#alanpearceYeah, I remembered it from somewhere, I guess
#KartikPrabhuthese negative aspects might be obvious to people who deal with them regularly, for beginners even a subjective POV on this helps (speaking from personal experience)
KevinMarks__ joined the channel
#tommorrisI think if we're to be a positive community, to make sure our rhetoric doesn't distract from the overall goal of encouraging people to build stuff.
#KartikPrabhutommorris: agreed. But that page has been there for a long long time and no one changed it until yesterday.
#tantekKartikPrabhu: oh we've discussed it repeatedly at HWC meetups in SF
#tantekand I documented it there as I saw a bunch of problems happening with database-backed indie web sites and didn't see *any* collection of such problems
#KartikPrabhutantek: I know it has been discussed extensively. but no one changed it. and that's what matters to outsider viewers
#tantekproblems that happened in *databases* not filesystems
#tantekthe "filesystem is a database" misses the specific points made in the article
#KartikPrabhutantek: of course. not saying you were wrong to write that.
#tantekKartikPrabhu: I collected a pattern of data points
#tantekthat's the part I think people are missing. or rather they're overreacting to.
#butteroh, ORACLE forked/bought your db, devs raged, forked a community verison which won't eat your conf files that are a 0.0.1 behind so the server didn't start?
#tantektommorris: I don't think the emo-tweeters are bothering to read that far.
#butterwhat's the slowverhead of using in-browser Filesystem-access feautres
#petermolnarMySQL is not a bad db; the MyISAM engine was, InnoDB is way ahead; also keep in mind that MySQL is for speed, not for exceptional stability, like Oracle
#tommorrisYep. Other people's lack of reading comprehension ceased to be my problem a long time ago. ;)
#butterhaving to plumb it all thru JS, browser security-layers , down into the OS
#petermolnarand don't forget that MySQL has different engines, not just one
#petermolnarin theory, we could write a flat-file engine for MySQL
#KartikPrabhupetermolnar: we could? as in store the actual data in files?
#butterclean-cut, stdlib functions in a scripting-lang are less fragile than sprintf()-ing SQL as well
#tommorris"The Blackhole engine is a no-op engine. Any operations performed on a table using Blackhole will have no effect. This should be born in mind when considering the behavior of primary key columns that auto increment." is possibly the greatest documentation anyone has ever written.
#KartikPrabhuthis whole war is strange to me, given the popularity of file-based "databases" like MongoDB
#alanpearceThat's document-based, which is different
#tommorrisKartikPrabhu: no, the really strange thing is how many people seem to use Jekyll and other static site generator tools
#KartikPrabhualanpearce: oh. I think I don't understand the difference then
#petermolnartommorris: the blackhole will not write anything locally but it will send the replication down, so the slaves can store the data; thus you can offload write to a slave, for example, for an archive table
#butterso /dev/null should be reimplemented anywhere that has IO and can't take a file-handle?
#tommorrispetermolnar: yep, I understand why it's there, I just like the tone the documentation is written in. Kinda Onion-like.
#alanpearceKartikPrabhu: yeah, the wording is a bit ambiguous. It's more like an 'object store': attributes and child objects are stored, but it's all binary JSON or similar.
#KartikPrabhualanpearce: so the diff. is in being a human-readable file format?
#LoqiHaxxa: finalcut left you a message 1 day ago: - you might not have url rewriting turned on in apache if you are getting a page not found error with the begin page. I was just able to recreate it by removing url rewriting
#rasculnot a spam bot but sometimes when there's a lot going on at #indieweb or on the wiki and little talking, it can look like it
#Haxxahmmm... whats your take on the indieweb rascul - what is your site based off? known - wordpress?
#bretcaught up on the logs. butter had me loling, do you have a website?
#tommorris"Host your own content. Don't let these silicon valley douchebags con you into giving your personal property to them to do with it as they please. If you want to post things on Youtube, Soundcloud, etc. Fine. But you should also host it on your own hosting account somewhere using a wordpress blog or whatever. Also, keep copies of everything in your physical
#tommorris"The Internet Archive is also a safe place to store media long term. I have been posting media on line since 1997 when I had a live Real Audio stream of Yeast Radio. The only place that hasn't deleted some or all of my content during that time is Internet Archive."
#tommorris"Silicon Valley doesn't give a fuck about you or your content or your rights."
#brettommorris oh yeah someone was asking someone in here on twitter if indiweb could help drag queens or something
#bretis there one too many "syndicate-to" key names in this string or is that a thing? syndicate-to=syndicate-to=twitter.com%2Faaronpk%2Cfacebook.com%2Faaronpk
ooland, dietrich, alexhart_, paulcp, eschnou, brianloveswords and npdoty joined the channel
#tantekkylewm what do you mean "when a p-name is nonsense"? can you provide an example URL?
#bearto be pendantic, urlparse.urlparse() that returns a tuple with the query as a string, it's urlparse.parse_qs() that returns a dictionary and yes, multiple items with the same name are returned as a list
#kylewmcontext of the parameter discussion is we were wondering if micropub should be using multiple entries for the same key instead of comma separated values
#tantek.comedited /manifesto (+326) "clarify no manifesto, add section header other manifestos, add Independents Day Manifesto as original art, note in summary no indieweb(camp) manifesto" (view diff)
#LoqiA manifesto is "a published verbal declaration of the intentions, motives, or views of the issuer, be it an individual, group, political party or government" according to Wikipedia[1] - in summary there is no IndieWeb(Camp) manifesto http://indiewebcamp.com/manifesto
#LoqiA manifesto is "a published verbal declaration of the intentions, motives, or views of the issuer, be it an individual, group, political party or government" according to Wikipedia[1] - in summary there is no IndieWeb(Camp) manifesto http://indiewebcamp.com/the_indiewebcamp_manifesto
#binbastii tend to agree when it comes to the inline link format, but that's nice to write. discourse does a great job of taking it as input, but transforming it to the more readable page bottom notation
#binbastii'd disagree with backticks (total unix) and headers
#bret.iocreated /gem (+166) "Created page with "{{ stub }} <dfn>Gem is the ruby language package manager.</dfn> It can be used to managed ruby packages and provide access the the packages at https://rubygems.org."" (view diff)
#binbastialthough your improved headline format already works
#kylewmthe wires get crossed for me that html has the URL first and markdown has it second
#kylewmso i often switch them when writing markdown
#tantekthe thing about HTML is that it has quite minimal use of punctuation: <..> and <../> to start with, then other parts are readable letters - tag and attribute names
#tantekI think that's the key - minimal punctuation that you have to commit to memory, and then use of text (which is easier to recall than random series of punctuation)
#tantekkylewm - MediaWiki also has URL first: [URL linktext]
#bret.iocreated /npm (+240) "Created page with "{{ stub }} <dfn>npm is the "node package manager" and is can be used to manage javascript package for any type of javascript project (not just node).</dfn> It provides quick ac..."" (view diff)
#binbastii think when you're writing it all day, typing less matters more to many people, which is why markdown and textile were invented in the first place
#tantekand I greatly prefer that because it uses only TWO punctuation characters [ ] as opposed to Markdown ()[]
#tantekbear, the thinking was to first design what would "make sense" and look reasonable in *plain text* and then see if it was something consistently parsabel
#Loqitantek meant to say: bear, the thinking was to first design what would "make sense" and look reasonable in *plain text* and then see if it was something consistently parseable
#bearoh agree completely - optimize for the part that you, the user, will be interacting with the most
#tantekbear - what's sad is that I think the crappy (line-noise) parts of markdown were design with parsing considerations first, and plain text considerations second.
#butteri agree with 95% of your wikipages, great decentralized/personal-sovereignty ethos, but the structured-data stuff as microformats? i'd take RDFa or microdata+Schema.org over that
#tantekbutter - why? do you like typing more than you need to? (RDFa microdata) and do you like trusting big companies with overdesigned data models? (schema)
#KartikPrabhuenter mandatory Volcano fax number reference
#KartikPrabhubutter: do you know of anyone (non-corporation) who parses/uses published RDFa or microdata to do anything?
#butteri prefer metadata in formats designed for it rather than messily embedded in HTML
#tantekexactly, why put extra effort or be biased towards just catering towards Google/MS/etc.
#butternot really a fan of RDFa either, but at least property-names are 3rd-party/decentralized-extensible not decided upon by a couple google/yandex employees
#KartikPrabhubutter: HTML classes are metadata anyway
#butterKartikPrabhu: most tools i use consumer Turtle, which has RDF inside. for a basic 'finder'/directory-browser https://github.com/deiu/warp , and so on
#KartikPrabhuand why is inserting classes messier than inserting "property" ?
#tantekthe funnything about all this single-page-site development fad is that once it's long over, it will be difficult to research/understand what the heck happened, because none of the single-page-site apps will have any resilient permalinks to view in the future.
#tantekit will be a sort-of blackhole period of JS development