#meta 2021-08-27

2021-08-27 UTC
rockorager and Seirdy joined the channel
#
Loqi Just generated the first draft of this week's newsletter! https://indieweb.org/this-week/2021-08-27.html I'll generate a draft again tomorrow, so please add to it before then! https://indieweb.org/this-week#How_to
jeremy and jeremycherfas joined the channel
#
@dreftymac "I've given up on using any sort of branded app for notetaking." via by @zachlatta @HackerNews #NoteTaking #Annotation #EdTech #IndieWeb #Zettlekasten https://news.ycombinator.com/item?id=15057002 (twitter.com/_/status/1431113280085319683)
[KevinMarks] and doosboox8 joined the channel
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
@schnarfed ↩️ Fun! Here's one: https://indiemap.org/ is a deep, structured sociological dataset of the @indiewebcamp community. 2300 web sites, 631M links, 706K relationships, and more. We haven't published on it directly yet, but https://dissertation.jackjamieson.net/ is highly related. #showmeyourdata (twitter.com/_/status/1431273788683948035)
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
@schnarfed ↩️ Another view of a small subset of the network with weighted edges. Interactive version is https://kumu.io/snarfed/indie-map (twitter.com/_/status/1431276776953581571)
#
capjamesg[d] @tell snarfed How long did it take you to crawl the sites for IndieMap?
#
capjamesg[d] Sorry... I meant...
#
capjamesg[d] !tell snarfed How long did it take you to crawl the sites for IndieMap?
#
Loqi Ok, I'll tell them that when I see them next
#
capjamesg[d] For fun, I changed a few lines of code on my website crawler to see if I could crawl other web pages. I sort of can (a bug needs fixed) but then I realised just how big sites are 😄
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
sknebel capjamesg[d]: afaik snarfed did that bit by bit over months, theres a video from one of the summits where he talked about it
#
sknebel capjamesg[d]: probably this one https://www.youtube.com/watch?v=bkc5afFohmI&index=5&list=PLk3TtIJ31hqrLIPqz55TczawWu-30cnXM
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
capjamesg[d] Thanks sknebel!
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
capjamesg[d] snarfed Did you make the IndieWeb crawl dataset public?
#
capjamesg[d] I'd love to ingest it into my blog search engine (separate instance) for NLP research purposes.
#
capjamesg[d] (I don't plan to make an IndieWeb search engine right now. Way too much work!)
#
capjamesg[d] Ah, found it!
#
[snarfed] capjamesg a couple months, on and off. https://indiemap.org/docs.html#stats
#
Loqi [snarfed]: capjamesg[d] left you a message 1 hour, 20 minutes ago: How long did it take you to crawl the sites for IndieMap?
#
[snarfed] (the "Count" y axis there is number of pages)
#
[snarfed] (and thanks sknebel!)
#
[snarfed] if you know SQL, BigQuery is the easiest way to poke around at the data, https://indiemap.org/docs.html#data-mining
[fluffy] joined the channel
#
capjamesg[d] [snarfed] I am hoping to use this to fine tune my search engine.
#
capjamesg[d] https://search.jamesg.blog
#
capjamesg[d] I have been implementing named entity recognition / direct answers lately (over weeks) but my site is only a few hundred pages large so my dataset is limited.
#
capjamesg[d] Thank you so much for doing this project. snarfed++
#
Loqi snarfed has 2 karma in this channel over the last year (49 in all channels)
#
[snarfed] welcome! glad you're interested
#
capjamesg[d] (The talk above was amazing!)
#
[snarfed] hah thanks! I enjoyed giving it
#
[snarfed] yeah if you're getting into more advance IR and search engine techniques, your dataset size will severely limit you. feel free to try building one on the indiemap dataset!
#
capjamesg[d] Yep! I'll keep you posted. I think I just found a project to eat my weekend 🙂
[aciccarello] joined the channel
#
capjamesg[d] Starting with snarfed.org now.
#
capjamesg[d] I have written a quick ingestion program so I can add the data through my crawler.
#
[snarfed] fwiw, crawling is a crazy tar pit of effort. just a subset of the challenges I hit and had to special case manually: https://indiemap.org/docs.html#exceptions
#
capjamesg[d] How many pages are on your site snarfed?
#
[snarfed] if you're more interested in the IR techniques (ranking etc), you might consider using the indiemap dataset instead
#
[snarfed] capjamesg 🤷 10k ish?
#
capjamesg[d] Sorry, I meant I am running your WARC file through the program that processes all HTML files in my crawler.
#
capjamesg[d] So not the actual crawl bit. Just the processing.
#
[snarfed] 10-50k?
#
[snarfed] ah ok!
#
[snarfed] :thumbsup:
#
capjamesg[d] This way I'll have all the data in exactly the way I need it to process according to my search algorithms.
#
capjamesg[d] (Quick win I am super proud of: I shaved off one second from a search query loading time. )
#
[snarfed] 👏
#
capjamesg[d] I just got to https://snarfed.org/majestic.
#
[snarfed] oh man short story writing, blast from the past
#
[snarfed] (we should probably take all of this to #indieweb or #indieweb-dev 😆 )
#
capjamesg[d] We should... Let's do it.
#
capjamesg[d] #indieweb-dev it is!
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
tracydurnell.com edited /personal_library (+140) "/* IndieWeb Examples */ add my reading list" (view diff)
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
[tantek] That guest blog definition could be improved a bit
#
[tantek] what is a guest blog
#
Loqi Guest blog is a practice of allowing another author to publish on your blog or website or publishing on someone else's website https://indieweb.org/Guest_blog
#
[tantek] grammar is a bit off
#
[tantek] also all the links / discussion about guest blog(ger(s)|ging) from #indieweb this past week is worth capturing in the page
#
[tantek] starting here: https://chat.indieweb.org/2021-08-24#t1629833548648600
#
Loqi [[jgmac1106]] tantek go read Alan Levine's posts on these messsages
#
[tantek] anyone around to help fill out that article before Newsletter deadline time so our one new page is more than just a dfn?
jjuran joined the channel
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
jjuran joined the channel
#
Loqi Generated a new draft of the newsletter! https://indieweb.org/this-week/2021-08-27.html
[jgmac1106] joined the channel
#
[jgmac1106] Sorry didn't see message [tantek] swamped at woek
#
Loqi Just generated this week's newsletter! You still have a few minutes to make changes, and I'll re-generate it 10 minutes before it gets sent out at 3pm Pacific time. https://indieweb.org/this-week/2021-08-27.html
#
Loqi Generated the final version of the newsletter! This will be sent out at 3pm Pacific time. https://indieweb.org/this-week/2021-08-27.html
#
@indiewebcamp This week in the #indieweb https://indieweb.org/this-week/2021-08-27.html (twitter.com/_/status/1431379764510539777)
#
@RobbiNespu Few days ago.. I removed #utterances from my website (https://robbinespu.gitlab.io/posts/commenting-system-utterances-and-disqus-removed-just-use-email/) and just now, i implemented indieweb… https://twitter.com/i/web/status/1431303404308111366 (twitter.com/_/status/1431303404308111366)
#
GWG Countdown to popup...
alex11 and Seirdy joined the channel
#
@BillSeitz ↩️ Howzabout micropub and markdown? https://indieweb.org/Micropub/Clients (twitter.com/_/status/1431400223776157697)
wackycity[d] joined the channel