#dev 2025-11-30

2025-11-30 UTC
#
burgeonlab <jeremycherfas> Oh, good idea to limit the original upload then to the (max possible future) display size then ;) Hugo has some good photo processing features but exif support isn't that great the last time I looked. I wanted to use it for filtering.
[schmarty] joined the channel
#
pcarrier in Create Day we briefly touched on counting visitors. approaches I can think to do that involve either a cookie or recording the client's IP. anybody know of/can think of decent heuristics without either?
#
aaronpk if the reason to avoid using cookies or recording the IP is to enable your visitors to remain anonymous, then you can’t really count them by definition, even if there was another mechanism
#
aaronpk for example, recording a hash of the IP has the benefit of never actually storing their IP address, but still being able to correlate future visits from that IP. so is that acceptable? I’m guessing no
#
pcarrier well I want to count them without personal data, if I had eg a random hash generated for each browser session and website that'd work out well
#
pcarrier IP hashes are not effective, most people use IPv4 so rainbow attack.
#
aaronpk that’s…not how rainbow attacks work
#
pcarrier I mean there are only 4B possibilities, that's trivial to reverse.
#
[morgan] Is it possible to clarify the purpose of counting visitors?
#
[morgan] Haha you have some wizard energy going on right now
#
aaronpk also who are you trying to protect against rainbow attacks from? …someone who accesses your logs of IP address hashes?
#
pcarrier aaronpk[d]: how is it not how rainbow attacks work? your input space is too small so you hash enough of it to find values for hashes.
#
[morgan] This isnt a direct answer but I use cloudflare and it tells me how many unique visitors
#
[morgan] maybe you are using it or could consider using it and offload the task
#
aaronpk who do you expect to reverse the hashes you are storing? yourself?
#
pcarrier morgan: yeah many people find it a more valuable insight than hits
#
pcarrier let's say myself yes.
#
aaronpk then i go back to my initial question, by definition you can’t count visitors without storing something that you can reverse engineer
#
pcarrier or someone who gets access to my "access logs"
#
pcarrier > well I want to count them without personal data, if I had eg a random hash generated for each browser session and website that'd work out well
#
pcarrier that would escape that definition.
#
pcarrier can't reverse engineer the hash to go back to someone's location or identity.
#
aaronpk where would you store that hash? in a cookie?
#
pcarrier browsers could do it for example.
#
aaronpk there are various formulas that let you approximate counts without being exact, maybe that would work? Redis has one built in
#
pcarrier how would redis help me distinguish between 1 user hitting 2 pages vs 2 users hitting 1 page each?
#
aaronpk https://redis.io/docs/latest/develop/data-types/probabilistic/hyperloglogs/
#
pcarrier (I use hyperloglog from redis for other stuff already)
#
pcarrier yeah I don't see how they help here
#
[morgan] Just so I understand - I dont think I can help here
#
aaronpk you can “add” the IP address to the set, but it’s never actually storing it
#
[morgan] you want unique IP addresses stored over a period of time
#
pcarrier the core problem is that I want a non-personal-data-correlated identifier
#
[morgan] so maybe someone hits your site a few times. or a bot does. But you just store a UNIQ ip
#
pcarrier that lets me track a "visit" or something akin to a visist
#
aaronpk anonymous traffic stats are even in their example for that command
#
pcarrier I specifically don't want to introduce a cookie myself (problematic in EU but also not a fan for a bunch of reasons), and I don't want to record IP addresses because that discloses too much about a person (specially if I have to reveal them to authorities)
#
aaronpk i’m pretty sure hyperloglog is your answer
#
pcarrier OK, thanks. will dive more into it, worried about reversibility in practice (specially at tiny scale)
#
[morgan] 🕵️‍♂️ Depends on the threat model I guess, good luck out their Agent Pierre etc. heh
#
aaronpk if you’re extra paranoid you can use a hash with a salt like the current day as the input to the HLL if you only care about visits per day, that would significantly reduce the possibility of using rainbow tables
#
pcarrier I'm generally extra-"paranoid" when it comes to the privacy of visitors yes 🙂 very conservative with what I track so far (https://xmit.dev/posts/analytics/) and I'd like to remain confident I don't risk leaking personal identification data. I mean if you compromise my systems you can tap into network traffic etc. so there's no risk 0, but might as well keep it as hard as possible to exflitrate historical data.
#
[morgan] Im sure you've considered it a lot, and Im impressed with your overall consideration and skill, but maybe not collecting the data might win out
#
pcarrier yup, that's where I'm landing for now, just wanna make sure I explore what can be done before giving up
#
[morgan] Ive often tried to explain to people how experiments and consideration like this is really good for professional growth even if the practicality is like, maybe no
#
[morgan] The last little exploration I went into was all about credit card autofill and trying to understand it. Have no practical or good purpose for it, just understanding how browsers autofill a little more
pfych joined the channel
#
pcarrier FWIW this is meant to be practical. the analytics work, they're just extremely limited (hits or bytes). examples at https://rrier.fr/@pc/statuses/01KB9BY7SZF8W9XAQFGDXTPCT8
grufwub joined the channel
#
[morgan] Oh interesting - you made a lot of this yourself ? The UI ?
#
pcarrier With tools yeah
#
[morgan] I guess Cloudflare unique visitors counter for a few websites I have on there tells me a few ambient things
#
[morgan] It's online, people are accessing it, it seems to spike when I post it or shared it a bunch
#
[morgan] But I have come to believe it doesn't do much more
#
pcarrier Hits will do that
#
pcarrier I'll proceed with what I have I guess. Might try to clean up the UI a bit (ideas welcome; only change since those screenshots are cleaner bookmarks hidden in <details>), but feature wise it's what I got
#
[morgan] It looks neat, good luck
#
[morgan] Like I think my comment about practicality - this is a window into skills that would be useful for a large org doing analytics - but if I gather right these are on some side projects that might not necessarily need it? All good time spent at any rate good luck
#
pcarrier Oh I've built an analytics product before, there wasn't much to learn here other than how to operate clickhouse on bare VPSes. This is just a few hours of putting something together for my hosting platform, that was high on the short list of features I thought it ought to have. I don't have a main project per se at the moment so not sure that qualifies as a side project, been quite focused on that hosting platform lately.
#
[morgan] 🧙‍♂️ good energy 💪
#
pcarrier The other reason to do it is that I now have a decent time series pipeline in place. I'm thinking about running backend services too, and it's essential to do that well (I'd need resource utilization metrics, logs, etc. there)
troojg, [fluffy], rgbivvv, marc_in_space, doesnm_, [KevinMarks] and ttybitnik joined the channel
#
[KevinMarks] [capjamesg] have a look at how bsky scales by having a sqlite db per user for their read feeds populated by a centralised crawler
#
[KevinMarks] So there's less write contention as it's reading a new feed, then writing to the sqlite for each user that is subscribed to it
#
capjamesg [KevinMarks] Do you have a relevant link I could look at?
#
capjamesg That sounds really interesting.
#
capjamesg I had to abandon the migration yesterday. I started by doing a dump of the DB to load into PostgreSQL but the disk ran out of space overnight. I then experimented with pgloader but for a 100GB+ DB it was going to take even more time and the process seemed to be going really slowly.
#
capjamesg If I could stick with SQLite that would be ideal.
PicoPixl joined the channel
#
[KevinMarks] Hm, looks like I remembered it backwards. They have a per user sqlite db for the PDS, not the viewed feed which relies on a shared CDN to expand a list of post uris. https://newsletter.systemdesign.one/p/how-does-bluesky-work
#
capjamesg [artlung] I made the change you suggested to the Validate Everything web page. thank you!
barnaby, bret, proofreed, DanQ, [Trevor_Morris], troojg, doesnm_, burgeonlab, capjamesg, tetov, nnrx, suki, hedy, eb, srushe, ancarda, athenaeryma, vikanezrimaya and rob32 joined the channel
#
[tantek] what's that web design error where a page autoloads more stuff when you scroll so you can briefly see but never actually see/click-on the footer? so annoying
[KevinMarks]1 joined the channel
#
[tantek] oh wow I outraced it with "end key" (fn ⬇️ ) and moving my mouse into position then clicking before the page could autoload more crap
jjuran_ and [snarfed] joined the channel
#
[tantek] two days late but I figure if retail can keep "Black Friday" going through the Monday after, then I can keep Create Day going through Monday too. added a /daylists page inspired by capjamesg++
#
Loqi capjamesg has 55 karma in this channel over the last year (212 in all channels)
#
[tantek] I'm also quite pleased with the minimal h-feed markup that gets parsed correctly: https://pin13.net/mf2/?url=https://tantek.com/daylists
#
[tantek] curious if anyone has a reader that consumes h-feed and what it looks like there
troojg joined the channel
#
pcarrier tankek: could also check the impedance mismatch with other formats through https://granary.io/
#
[tantek] what impedance mismatch?
#
[tantek] this is more about how robust are readers built to handle simple(r) content
#
pcarrier I don't know, I assume with data formats conversions aren't perfect, could be perfect here.
#
[tantek] we've done a decent amount of homework here 🙂
#
[tantek] perhaps I do need to publish my feed formats timeline summary
#
pcarrier well so https://granary.io/url?input=html&output=rss&url=https%3A%2F%2Ftantek.com%2Fdaylists loses authorship information for example
#
pcarrier IDK if RSS even has authorship data, if it's at the feed or post level or both, but that's the sort of stuff I'm talking about
#
[tantek] it does, so you might be running into a granary issue. see related https://github.com/snarfed/granary/issues/195
#
Loqi [preview] [alexmingoia] #195 Figure out mf2 h-feed authorship
bterry1, CrimeWave, boramalper, to2ds, barnaby and DanQ joined the channel