#dev 2022-09-26
2022-09-26 UTC
mcint, geoffo, gRegorLove_, gRegorLove__, [jgmac1106], ekhabarov and [Murray] joined the channel
# 
[Murray] Relevant dive into the various methods of storage on the web: https://web.dev/storage-for-the-web/
zack[m], jordemort, h4kor[m], mpardalos[m], Pyroxtheythem[m], tetov-irc, petermolnar, bterry and geoffo joined the channel; petermolnar left the channel
# 
jordemort finally finished my client-side search: https://jordemort.dev/blog/client-side-search/ - in a nutshell, i build a sqlite database with full-text index of all my pages at site build time, and then your browser uses sql.js-httpvfs to do byte-range requests on that database to query it
# jordemort dunno if anyone else is interested, but if we could get a group of folks to agree on a database schema, and a <link rel=...> to specify the location of the database for each site, it could be possible to build some sort of federated search on this
# [snarfed]1 jordemort out of curiosity, how big is the index? on many personal sites it's often small enough to just download the whole thing client side
# jordemort currently about 184k
# [snarfed]1 exactly!
# [snarfed]1 re federated, one big problem is scaling. making search requests to every single site will bog down once you get into the high hundreds or so
# [snarfed]1 we already see this on eg posts with lots of wms that make the browser download each profile picture from its own site
# [snarfed]1 https://indieweb.org/chat-names is another example
# [snarfed]1 and those are at least progressively usable. search, less so, since you'd need to rank all of the results globally
# [snarfed]1 (which also begs the cross-site ranking question 😁)
# jordemort i was envisioning a usage more like a feed reader, where each person would have their own curated selection of sites that they were interested in searching
# jordemort not trying to build indieweb google
# [snarfed]1 ah! like custom search engines in the browser
# [snarfed]1 that makes much more sense
# jordemort although if you wanted to you could probably spider the sqlite databases and store cached copies of them locally; you wouldn't actually be under the obligation to use byte-range requests or to implement the search client-side
# [snarfed]1 true!
# [snarfed]1 crawling HTML and browser custom search engines are both pretty compelling existing solutions to this
# [snarfed]1 HTML maybe arguable, but browser custom search engines, definitely
# jordemort the core of the idea is "discoverable sqlite database with full-text index of a site in a specific schema" and then what to build on top of that is more of an open-ended question 😄
# [snarfed]1 custom saearch storing a single search URL pattern per site is way easier for adoption, you don't need sites to do anything differently
# [snarfed]1 oof. understood! could be a fun building block. we tend to start with existing problems first here, but never hurts to try it out on your own site!
# [snarfed]1 unrelated, Bridgy Micropub docs are up: https://brid.gy/about#micropub
# jordemort i'm up for developing the idea further and putting together a demo of searching multiple sites at once, if anyone else thinks it would be cool or interesting or has ideas or other potential applications, but if i'm the only one excited about it i probably won't bother and will just be happy with the search i built for my site
# [snarfed]1 sounds like the right approach!
# jordemort i wouldn't want to go much further with it without getting some other folks' takes on the database schema and what should go in there and what should not
# [snarfed]1 definitely check out prior art, eg https://en.wikipedia.org/wiki/OpenSearch , Elastic, Lucene, Solr, Nutch, etc
mro joined the channel
# jordemort i have 🙂
# [snarfed]1 :thumbsup:
jacky joined the channel
# [snarfed]1 anyone here with any security auditing experience? Micropub is the first Bridgy feature that exposes private data (silo tokens) to end users. I put it behind silo auth, but I'd love someone else's eyes on it to see if I got anything wrong!
# [snarfed]1 code is https://github.com/snarfed/bridgy/blob/02e55f4132a67ad9581412e19ff6558f94b59c10/micropub.py#L188-L228 . it uses https://flask.palletsprojects.com/en/2.2.x/api/#flask.flash to serve and render the token, which temporarily stores it in a cookie
# [snarfed]1 ... (secure, same site) until it's renderd
Seirdy joined the channel
# 
aaronpk lol so helpful flickr https://www.flickr.com/services/api/flickr.push.subscribe.html
mro joined the channel
# aaronpk so, i have a bunch of code that crawls a user's flickr account and downloads everything to local files including a bunch of metadata. i now want to turn that into a static site that works by either opening it on the filesystem or deploying it to a web server. so basically what i'm after is a static site generator
[aciccarello] joined the channel
# 
[aciccarello] 11ty provides a lot out of the box and has lots of customizability. I use it for my site and would recommend it for a simple site.
# [aciccarello] Sure
# [aciccarello] If you have JSON files, it shouldn't be too difficult to write a liquid/nunchucks template for it. Pagination is the key thing for looping through data
# [aciccarello] Here's an example of the template for my tags page for reference. https://github.com/aciccarello/ciccarello.me/blob/main/posts/tags/tagPage.liquid
# [aciccarello] Ah, so a set of folders with json/jpg files 🤔
# [aciccarello] My first thought would be to throw that in the _data folder and then work from there. They key thing is keeping all the metadata related to the photo easily accessible.
# [aciccarello] Honestly a single json file with an array of photo metatdata objects would be easiest to work with 😅
# 
[KevinMarks] Hugo will likely make short work of it too, it comes down to which template language you like best.
# [aciccarello] Yeah, you might be able to write a script to do that quicker
gRegorLove_ joined the channel
tetov-irc, [tw2113_Slack_] and geoffo joined the channel