#social 2017-10-25

2017-10-25 UTC
#
cwebber
whether or not there's a problem with a continuation barrier isn't even deterministic
#
cwebber
or not within parameters I know how to control
#
cwebber
I'm not having a problem against my own test server but I am against puckipedia's ... which, it's definitely Guile's (or my) fault somehow, but I don't know how
#
cwebber
it's probabalistic here... I run that piece of code several times and it fails sometimes and not others
#
puckipedia
hm. that's the https one?
#
cwebber
nope, the http
#
cwebber
one
#
cwebber
this is what's causing it to "freeze" up
#
puckipedia
huh
#
puckipedia
anyways, ehm, that server is going down
#
cwebber
puckipedia: actually, let's try it against the https one
#
puckipedia
feel free to make an account on kroeg.azurewebsites.net
#
cwebber
where do I do that?
#
puckipedia
it does nothing with the email
#
puckipedia
it most definitely is valid though :P
#
puckipedia
oh huh
#
puckipedia
it doesn't do checking, so it needs the oauth parameters
#
puckipedia
response_type=token, redirect_uri=whatever (will append fragment)
#
puckipedia
it actually wants a state too
#
puckipedia
anyways
#
puckipedia
gnight :P
#
cwebber
gnite puckipedia
#
cwebber
OH
#
cwebber
could it be the call to FCNTL even when it isn't https?
#
cwebber
wait no that wouldn't be it
#
cwebber
I have no idea.
timbl, rowan and xmpp-social joined the channel
#
ajordan
testing the bridge?
#
xmpp-social
[ajordan] Excellent
xmpp-social, dlehn, rowan and timbl joined the channel
#
cwebber
hi
#
aaronpk
Morning
#
puckipedia
I moved my entire site to ActivityPub :P still have to migrate content though
#
cwebber
puckipedia: cool!
#
puckipedia
also /lots/ of debugging to do
#
cwebber
I suppose we have a SocialCG meeting in an hour, eh?
#
puckipedia
good point
#
cwebber
btw, just in case someone wanted to verify thta it does run under https now
#
cwebber
https://test.activitypub.rocks/ try with user https://chaos.dustycloud.org/u/tester and JNbn70qGRGTlYm8ZHPmVzQwtfj5ahAHnz9JVpwsy
#
cwebber
I think I developed a hunch as to where (though not why) the continuation barrier things were happening...
#
cwebber
puckipedia: it may actually not happen over https on your site now that https is working, I'm not sure. I should give it a try now I suppose
#
cwebber
I'm hopping in the shower and etc now though
#
puckipedia
right
#
puckipedia
ouch
#
cwebber
puckipedia: ouch?
#
puckipedia
it's not doing very well on the tests :P
#
cwebber
oh, did it run further than it did before?
#
puckipedia
yep, it got all the way to "Fetching the inbox" manual test
#
cwebber
well that's better, though I'm guessing you don't know why the tests are all failing
#
puckipedia
yeah, gonna run it against my local instance with some debugging turned on
#
cwebber
if things aren't hitting that bug that frees me up to add the debuggging stuff your equested
#
puckipedia
ah, that's also good :P
#
puckipedia
it'd be nice to see why it gives an unknown server error
#
cwebber
that one's probably a bug on my end
#
cwebber
but here it is
#
cwebber
oh it looks like it got html back
#
cwebber
and tried to read it as an as2 object
#
cwebber
I guess it should warn the user about that
#
puckipedia
yep. it threw a 403
rowan, rowan_ and alanz joined the channel
#
puckipedia
joins socialcg mumble
#
xmpp-social
[ajordan] Whew okay
#
cwebber
too
#
xmpp-social
[ajordan] Meeting?
#
nightpool
waves
#
cwebber
aaronpk, you joining us? :)
#
aaronpk
yes! one sec
#
nightpool
cwebber: just happened to see your post on masto, decided I had time to jump in and felt guilty for not having been to a meeting recently
#
nightpool
:D
#
cwebber
trackbot, start meeting
#
trackbot
is preparing a teleconference.
RRSAgent joined the channel
#
trackbot
RRSAgent, make logs public
Zakim joined the channel
#
RRSAgent
I have made the request, trackbot
#
trackbot
Meeting: Social Web Working Group Teleconference
#
trackbot
Date: 25 October 2017
#
cwebber
scribenick: aaronpk
#
cwebber
present+
#
aaronpk
present+
#
puckipedia
present+
#
nightpool
present+
#
alanz
present+
#
ajordan
present+
#
aaronpk
cwebber: most of the topics today are a holdover from last week
#
aaronpk
TOPIC: Social WG Updates
#
aaronpk
cwebber: the activitypub test suite is live
#
aaronpk
... but still hammering out the bugs with puckipedia
#
aaronpk
... so that will take at least the rest of the day before announcing it publicly
#
aaronpk
... we are hoping to go to PR by the 14th of next month
#
aaronpk
... that means getting the test suite really done, doing all the planned edits, and getting implementation reports in
#
aaronpk
... aaron there was something about websub
#
cwebber
aaronpk: we talked about a couple of minor editorial edits, and someone submitted a full review, but also yesterday we got a new implementation in the wild... twitch.tv has a new webhook platform using websub so that's exciting
#
Zakim
sees ajordan on the speaker queue
#
cwebber
ack ajordan
#
Zakim
sees no one on the speaker queue
#
aaronpk
ajordan: ryan barrett launched fed.brid.gy and that has successfully been used to reply from an indieweb site to a mastodon post
#
aaronpk
so that seems to be working now which is phenomenal
#
ajordan
fed.brid.gy
#
cwebber
csarven: oops
#
aaronpk
nightpool: they haven't quite replied yet because it doesn't show up as in reply to but the post itself did federate
#
cwebber
csarven: no idea what happened will look post-meeting
#
cwebber
csarven: this version is bridging to my own local testdev instance for just the moment, heh
#
aaronpk
TOPIC: media upload as extension
#
aaronpk
cwebber: the mediaUpload is moving to the CG to standardize as an extension
#
aaronpk
... we need to start the process
#
aaronpk
... iguess that's on me unless someone wants to step up
#
aaronpk
nightpool: what does the process look like for extensions?
#
Zakim
sees ajordan on the speaker queue
#
aaronpk
cwebber: i think it's still somewhat ad hoc
#
rhiaro
icymi I ripped media upload out and put it on the cg wiki yesterday https://www.w3.org/wiki/SocialCG/ActivityPub/MediaUpload
#
aaronpk
... someone ends up writing it up in this group and brings it to a meeting and we discuss it and ratify it as an official document through a vote
#
cwebber
ack ajordan
#
Zakim
sees no one on the speaker queue
#
cwebber
rhiaro, yay
#
aaronpk
ajordan: i might be willing to work on a document or two in a couple weeks
#
aaronpk
... if noone else volunteers i might be able to do that maybe question mark
#
aaronpk
cwebber: you were also looking at activitypub implementation in pump.io?
#
aaronpk
ajordan: yeah that's one of the things i need to do in the next couple weeks before i commit to anything else
#
aaronpk
cwebber: amy on IRC points out she moved the media upload section to the wiki
#
cwebber
q?
#
Zakim
sees ajordan on the speaker queue
#
Zakim
sees ajordan on the speaker queue
#
cwebber
ack ajordan
#
Zakim
sees no one on the speaker queue
#
aaronpk
ajordan: what would we need to be prepared to have a discussion on that?
#
aaronpk
... we want to look at other implementations like twitter?
#
ajordan
basically what are our action items here
#
aaronpk
cwebber: here's the challenges the media upload endpoint was hitting
#
aaronpk
... creating an activitystreams object and uploading the media in one swoop which might be fine but one challenge is you wouldn't want to do the create at the same time if you were going to attach it to something else
er1n joined the channel
#
aaronpk
... puckipedia and i came up with something which is whethre you wrap it in create is whether it shows up in your outbox
#
aaronpk
... but that didn't take into account chunked uploads
#
ajordan
thanks
#
er1n
o/
#
cwebber
q
#
cwebber
q?
#
Zakim
sees no one on the speaker queue
#
aaronpk
TOPIC: follower migration
#
Loqi
[sandhawke] #1 Follower Migration
#
aaronpk
cwebber: this issue has a lot of discussion
#
aaronpk
... a number of different routes have been discussed
#
aaronpk
... one would be to have some sort of "move" activity
#
Loqi
[Gargron] ```js { type: 'Move', actor: 'uri/to/alice', object: 'uri/to/alice', target: 'new_uri/to/alice' } ``` Sent out to followers. Old server's webfinger would need to redirect to new server's webfinger, and new server's webfinger woul...
#
aaronpk
... why not codify it as an AS2 object to move one user to another
#
aaronpk
... one downside is it wouldn't preserve the URIs of the old activities
#
aaronpk
... one reason mastodon didn't do this is it would allow someone to take a big action all at once. but now mastodon has the option to delete an entire user all at once.
#
jaywink
FYI. here is the diaspora protocol account migration entity, for reference: https://diaspora.github.io/diaspora_federation/entities/account_migration.html
#
aaronpk
cwebber: another route would be to have some sort of decentralized identifier
#
aaronpk
... DIDs are a combination of replacing DNS and SSL authorities at the same time
#
cwebber
q?
#
Zakim
sees no one on the speaker queue
#
aaronpk
sorry i missed a lot of that
#
aaronpk
... another option is zot's version of nomadic identity
#
aaronpk
... which i don't entirely follow. i did take a look at it at one point and when you boil it down it is namespacing a combination of what's effectively the domain it's posted on and someone's unique key
#
nightpool
(to close the loop async: msatodon does parse a Delete { Actor } as deleting an entire actor and all of their content, but doesn't expose this to users in any way))
#
aaronpk
... the goal is if you make a post the post has an ID and even if the server is down the post still is identifiable
er11n joined the channel
#
aaronpk
cwebber: so, the last one is HTTP redirects
#
aaronpk
... which could be combined with a move activity
#
aaronpk
... maybe you do the move activity and set up http redirects on the old objects
#
puckipedia
q+
#
Zakim
sees puckipedia on the speaker queue
#
cwebber
ack puckipedia
#
Zakim
sees no one on the speaker queue
#
aaronpk
puckipedia: i've been playing with activitypub, and i was thinking of storing the objects as a hash of the object but then you get immutability. but i did take a look at the decentralized IDs and the question is are one of those enough to find the user or object?
#
aaronpk
... if you can't take an object ID and find its data then that's kind of the issue with migration
#
aaronpk
cwebber: in this paper i effectively show how you end up mapping a decentralized identifier to an activitypub object
#
aaronpk
... you should be able to use DIDs in one of two ways.
#
aaronpk
... you can replace the objects with DIDs. for example one uses IPFS, another uses bitcoin blockchain, another uses a specific blockchain
#
aaronpk
... so if you lose your phone, you can say if 2 out of these 3 people say it's okay then you can switch to the new key
#
aaronpk
... so the example in there shows an activitypub service, it links you to the activitystreams object for the user. but you can also link to it itself
#
aaronpk
... so if the receipient is something like this:
#
cwebber
{"to": ["did:example:d20Hg0teN72oFeo0iNYrblwqt#services/ActivityPub"]}
#
aaronpk
... you could resolve the object and pull down the activitystreams actor
#
aaronpk
... there's also a mechanism to do something like HTTP get and post but that's a bit underspecified
#
aaronpk
... so there is a way to be able to resolve the object with the DID itself.
#
nightpool
q+
#
Zakim
sees nightpool on the speaker queue
#
aaronpk
... i think there's a way to move to DIDs instead of domain names
#
aaronpk
... but there's not a clear path to moving from domain names to DIDs. it's clearer if everything uses DIDs from the start
#
aaronpk
puckipedia: maybe we could define a property like "alternate IDs"
#
aaronpk
cwebber: i'd need to think more of the consequences of that
#
cwebber
ack nightpool
#
Zakim
sees no one on the speaker queue
#
aaronpk
nightpool: i'm trying to follow along. can you give a 10000 foot overview of how resolution of IDs work?
#
aaronpk
... i remember some past discussions there was some concern over how web-like these mechanisms are
#
aaronpk
cwebber: there are multiple methods of resolution for DIDs, each specifies its own
#
aaronpk
... in the main DID protocol we're not sure which of the mechanisms are going to work out. the IPFS is probably the most promising
#
aaronpk
... the bitcoin one has a lot of overhead. the other one has probably the least overhead
#
aaronpk
... but the different methods need to specify the behavior of how to resolve
#
nightpool
can someone link the veris(sp?) protocol? googling doesn't seem to be turning anything up
#
aaronpk
cwebber: there are multiple DID methods. bitcoin, ipfs, veris. they each provide a create, retrieval and update method.
#
aaronpk
... however, some of those may require pulling down a large amount of information, such as bitcoin requiring pulling the whole bitcoin graph
#
aaronpk
... so similar to how people don't host the whole DNS database, you can have resolvers doing resolution as a proxy
#
aaronpk
nightpool: right now existing implementations are based around HTTPS, so moving from that to a system that's more complicated seems to be a hard ask
#
aaronpk
cwebber: so we're exploring multiple methods of follower migration. HTTPS methods require the previous instance be online
#
aaronpk
... if there's no redirects then none of the servers will know the content moved
#
aaronpk
... if you have a DID then it's more resilient if a server goes down
#
aaronpk
... so how will be implement the resolvers
#
aaronpk
... i think you just have a library and point at one or more resolvers
#
nightpool
q+
#
Zakim
sees nightpool on the speaker queue
#
aaronpk
... you'd be able to say get this DID and point at the resolver you want
#
aaronpk
... so we're presuming that it's unlikely that every mastodon server will host a DID database
#
cwebber
ack nightpool
#
Zakim
sees no one on the speaker queue
#
aaronpk
nightpool: that brought some clarity but also raised more questions
#
aaronpk
... wer'e talking about these DIDs like they're completely interchangeable. but all of them have vastly different drawbacks and affordances.
#
aaronpk
... it seems like from an implementation level we can't just push it to a library and have it do the right thing
#
aaronpk
cwebber: when i was talking about the hub and resolvers, i was talking about a resolver that handles all the methods for you under the same interface
#
cwebber
q?
#
Zakim
sees no one on the speaker queue
#
aaronpk
cwebber: where do we go from here? any comments?
#
Zakim
sees aaronpk on the speaker queue
#
cwebber
ack aaronpk
#
Zakim
sees no one on the speaker queue
#
cwebber
scribenick: aaronpk
#
cwebber
aaronpk: this feels like going down quite a rabbit hole... I'm a bit confused because everything going around this point has been based around HTTP and very web-like protocols, and up until this point we've been talking about very web-like things
#
ajordan
aaronpk: sounds like kinda what you're saying is, are we making the right tradeoffs?
#
cwebber
aaronpk: I realize DNS can go down and you're stuck, but you're very likely to have issues with these other systems and we might not know what this is. one benefit of https is dns is not tied to a server, you can move it to a new server etc without having to change your identity... so it seems like http/dns are very natural fits for this, esp in the context of the web, this is the social web community group
#
Zakim
sees ajordan on the speaker queue
#
aaronpk
cwebber: i think you're right, https is well understood and widely implemented. it depends on what definition of the web you're talking about.
#
aaronpk
... and you're also right that these are kind of out-there tehcnologies that haven't proven themselves yet
#
aaronpk
... i'm not saying these are things we should necessarily do, just trying to look at paths to explore
#
aaronpk
... and trying to find patterns that map more closely to the things we use
#
cwebber
ack ajordan
#
Zakim
sees no one on the speaker queue
#
aaronpk
... so i'm saying here are some paths we coud potentially go down
#
aaronpk
ajordan: obviously we dont want to end up with a bunch of different ways to do follower migration. but it sounds like some of the things we're discussing here are breaking changes anyway.
#
aaronpk
... so if we start addressing everything with DIDs then it sounds like that would already be a breaking change in the sense that any server on the network that couldn't resolve those addresses would be screwed
#
aaronpk
... so i'm wondering can we do something now, and explore these farther out options later
#
aaronpk
cwebber: that makes sense, that's what i was hoping to push here. get a discussion of what kind of thing we can do now and what can we look for in the future.
#
cwebber
q?
#
Zakim
sees no one on the speaker queue
#
aaronpk
ajordan: so we can do things in parallel too. we can ship a minimum viable solution, like shipping http redirects.
#
aaronpk
... with the assumption that the far-out things would take longer since we would have to build more stuff and also wait for underlying specs to mature as well
#
cwebber
q?
#
Zakim
sees no one on the speaker queue
#
aaronpk
cwebber: so now it's a matter of what people are interested in implementing right now
#
nightpool
q+ PurgeActor
#
Zakim
sees PurgeActor on the speaker queue
#
aaronpk
... it might end up being that the best way to continue this discussion is to coordinate if some of these paths start to get implemented
#
cwebber
ack nightpool
#
Zakim
sees PurgeActor on the speaker queue
#
ajordan
ack PurgeActor
#
Zakim
sees no one on the speaker queue
#
aaronpk
nightpool: when we talked about this on the mastodon issue about how to delete a user, you mentioned some kind of extension type that specifies side effects like migrating content
#
aaronpk
... and in the case of deleting a user should that delete all of the activities or should it just make the user inaccessible
#
aaronpk
... maybe the way forward here is thinking about extensions in that area
#
aaronpk
... so you could move an actor and say to also rewrite all the IDs
#
cwebber
q+ to talk about rewriting ids
#
Zakim
sees cwebber on the speaker queue
#
aaronpk
... or you can move an actor and it will just change the representation of the actor in the system
#
cwebber
ack cwebber
#
Zakim
cwebber, you wanted to talk about rewriting ids
#
Zakim
sees no one on the speaker queue
#
aaronpk
cwebber: what you brought up of what if the move activity also provided some sort of rule to rewrite IDs. i'm a little worried it will be hard to get right
#
aaronpk
... reason 1 is you could imagine someone writes a regex to rewrite from one system to another. regexes are pretty messy
#
aaronpk
... but also it could open a vector of attack, you could end up saying move all my posts and also gargron's. so i'm nervous about the template rewriting thing. but you're right that if we get this to work we need to figure out howto get it to work correctly
#
ajordan
I don't think you'd need regex-level template rewriting
#
nightpool
rewriting IDs from something like https://example.org/ID to something like https://new.server/authority/https://example.org/ID is probably doable
#
ajordan
I think we might be able to just get away with specifying a less powerful system and then you avoid all those vulns
#
aaronpk
cwebber: yes specifying a less powerful system would be good. maybe we can start by writing a wiki page about it
#
aaronpk
nightpool: if you're thinking about some sort of template thing, you can hold the signed move activity and any server that missed it can verify it using linked data signatures
#
aaronpk
... so if you have an authentication protocol like that in place then you dodged the "oh the server has to be up" issue
#
aaronpk
... because any server can have a signed activity and send it to another server
#
aaronpk
cwebber: so if the actor has the same key as they had on the previous server, you can be sure it's the same user?
#
aaronpk
nightpool: yeah if you have the key for nightpool@cyber.space you could import that into a new server and let me sign things as nightpool@cyber.space
#
aaronpk
... so maybe this is a little out of spec since mastodon currently thinks of keys as authoritative and there isn't guidance on that in activitypub, we resolve keys through webfinger and map them to users that way
#
aaronpk
... i'm not really sureh ow linked data signatures fit into the spec as written
#
aaronpk
... but if i can prove i'm acting on behalf of an actor then i don't need to worry about the decentralized ID tihng
#
aaronpk
cwebber: so mastodon does attach the public key to actors. so if you have a cached copy of the key on an actor and you're able to demonstrate you have the key, then you can do it even though you can't do webfinger requests anymore
#
aaronpk
did i get that right lol
#
aaronpk
cwebber: also users would have to be able to download the key associated with their profile so that they can upload it to the new server when they move over
#
nightpool
+1 for deferring, but maybe give a quick summary?
#
ajordan
I cannot extend
#
aaronpk
can't extend
#
aaronpk
TOPIC: publishing extensions
#
aaronpk
cwebber: currently in activitypub, there is an extension mechanism for vocabularies. you can have a way to specify new vocabularies not defined in activitypub, but you don't have a mechanism to be sure that the side effects of the activities will be understood by the other servert.
#
aaronpk
... so that's what this proposal is about, defining what type sof activities your server is able to understand the side effects for
#
aaronpk
... the extension provides a new endpoint that gives back a list of all the extensions represented by URIs that I know how to handle
#
ajordan
I herd u liek extensions
#
cwebber
haha
#
Loqi
rofl
#
nightpool
sgtm
#
ajordan
so I put some extensions in your extensions so you can extension while you extension
#
ajordan
aaronpk++ for scribing!
#
Loqi
aaronpk has 96 karma in this channel (1450 overall)
#
cwebber
aaronpk++
#
Loqi
aaronpk has 97 karma in this channel (1451 overall)
#
nightpool
aaronpk++
#
Loqi
aaronpk has 98 karma in this channel (1452 overall)
#
ajordan
cwebber++ for chairing too
#
Loqi
slow down!
#
aaronpk
trackbot: end meeting
#
trackbot
is ending a teleconference.
#
trackbot
Zakim, list attendees
#
Zakim
As of this point the attendees have been cwebber, aaronpk, puckipedia, nightpool, alanz, ajordan
#
ajordan
cwebber++ for chairing too
#
trackbot
RRSAgent, please draft minutes
#
RRSAgent
I have made the request to generate http://www.w3.org/2017/10/25-social-minutes.html trackbot
#
trackbot
RRSAgent, bye
#
RRSAgent
I see no action items
#
ajordan
Loqi: you're letting me down :(
rowan and tantek joined the channel
#
tantek
just got back online
#
cwebber
oh no wonder
#
cwebber
csarven: it's back up
#
xmpp-social
[ajordan] tantek: o/
#
cwebber
https://test.activitypub.rocks/ is being port-forwarded from my machine via ssh
#
cwebber
I can run it on the server for real but
#
cwebber
I have it local so I can do debugging while working with puckipedia
#
cwebber
the ssh connection died... guess autossh didn't automatically bring it back up
#
cwebber
runs `cmatrix -u 10' to keep the ssh connection from timing out from inactivity.. that should do it, right? ;)
#
nightpool
cwebber: just so I'm clear, the test suite doesn't have any S2S stuff yet, right?
#
tantek
I thought it did
#
tantek
pretty sure because snarfed (author of fed.brid.gy) remarked that he was happy to see it
#
Zakim
excuses himself; his presence no longer seems to be needed
#
Loqi
yeah who invited you anyway Zakim
#
alanz
Is Zakim a bot?
timbl_ and rowan joined the channel
#
xmpp-social
[ajordan] Yep!
#
alanz
quite funky, is it open source, available somewhere?
#
alanz
found it, nvm
#
puckipedia
the first actual other person followed me from Mastodon towards my Kroeg instance! and, well, https://dl.puckipedia.com/up/companioncube.png
#
puckipedia
bit later, I got it working nicely: https://dl.puckipedia.com/up/woo_avatars.png
#
puckipedia
is happy
#
cwebber
nightpool: it has S2S kind of "prompt'y" tests where it asks you to try things, and then describes the behavior you should see and you check if it's correct
#
cwebber
in the future it may do it in a more automated fashion
#
cwebber
the SocialWG agreed that since we have real interoperable implementations this was good enough for us to get to PR
#
cwebber
I'd still like the automated tests tho
#
cwebber
now to get back to cleaning some stuff up... gonna add the debugging feature puckipedia asked for and also fix the websocket timeout issue
#
puckipedia
yay :)
#
cwebber
nightpool: aaronpk: btw I still ack that it's another layer to add, but one thing I forgot to add is, depending on the http
#
cwebber
er
#
cwebber
on the did method you use, it may be http
#
cwebber
veres one is, and iirc there are ipfs network layers that are
#
cwebber
but definitely that's not true for bitcoin.
#
cwebber
also true probably if using a resolver
#
cwebber
nonetheless even if that's true ack that it's a new abstraction layer to absorb
eugr joined the channel
#
eugr
hm
#
eugr
hey guys
#
eugr
so i'm working on cobalt, my take on mediagoblin territory. i'm still a bit unsure about how to schema the storage for AP stuff
#
cwebber
hi hi eugr
#
cwebber
want my opinion on how I would have done it if I started mediagoblin over?
#
eugr
how have y'all done it on your end? i want to try a different approach to mastodon's full normalization
#
eugr
like, it was such a bother in masto because AP payloads had to be generated on the fly
#
eugr
so right now i am considering an activities table (account_id:bigint, payload:json)
#
eugr
in postgres ofc
#
cwebber
eugr: yeah using jsonb I'm guessing?
#
eugr
yeah
#
cwebber
eugr: you'll also want an opportunity for private fields
#
puckipedia
my plan is to use PostgreSQL as a huge tuple store, but currently I have a table { entity_id: string, is_owner: boolean, payload: jsonb, type: string }
#
eugr
wait is json and jsonb different things?
#
puckipedia
eugr: jsonb uses binary json internally
#
cwebber
eugr: jsonb is waaay better than json
#
cwebber
b-trees
#
eugr
oh okay
#
cwebber
eugr: as for the private fields, eg a password
#
eugr
what private fields?
#
cwebber
password hash
#
puckipedia
cwebber: tbh, I fixed that by using an out-of-band mechanism for that
#
eugr
um no why tho. it's just for activities, no?
#
eugr
i still have an accounts table
#
puckipedia
eugr: I store *everything* in the object store. each Person is in the store as well
#
eugr
i don't think i want to do that
#
puckipedia
actually, the Person gets Created by itself as well :P
#
cwebber
eugr: ok, what I was going to say is that I'd link in other tables to each of the linked data/ as2 objects depending on their type
#
cwebber
so for example an Account table may link to the Person recrod
#
cwebber
record
#
cwebber
and hold some of the "common" field
#
cwebber
s
#
puckipedia
eugr: you're still going to have to be able to process external users
#
eugr
hm
#
cwebber
eugr: then some of the data you expand dynamically
#
cwebber
eugr: here's the other major thing I'd do
#
cwebber
eugr: you actually want to do content based addressing of objects
#
eugr
oh boy it's still a mess, no matter which approach you take... normalization at least made a lot of sense to me at the start
#
eugr
okay so
#
puckipedia
also my suggestion: expand/re-compact with a specific AP context when storing in the DB
#
eugr
table "objects" (uri:string payload:jsonb)??
#
cwebber
eugr: I'd instead do this
#
puckipedia
eugr: if you want to be able to cache external objects, also a flag is_owner :P
#
eugr
what is is_owner?
#
eugr
is that like local:boolean?
#
puckipedia
yep
#
eugr
i would call it local :P
#
puckipedia
so you know what you can evict :P
#
cwebber
(uri:string cas_object:cas_hash) -> (cas_hash, payload:json)
#
puckipedia
eugr: tomato, tomato :P
#
eugr
cas_object? wut
#
cwebber
eugr: so think of git
#
cwebber
objects are immutable hashes
#
puckipedia
hmmm
#
eugr
eh
#
cwebber
but each branch points at an object
#
cwebber
so there's a reason for this
#
eugr
git repos get really big over time tho exactly for this reason. why would i store outdated versions of objects?
#
puckipedia
... it's signatures, isn't it
#
Loqi
[msporny] > {"object": {"object": {"object": {}}}} -1, don't do this; the approach will lead to sadness and heartache. Keep in mind that any of those objects could change at any time in the database. If you're signing objects AND deconstructing those obj...
#
puckipedia
bingo
#
eugr
i find this hard to process
#
eugr
:/
#
puckipedia
eugr: basically, you can deconstruct incoming objects, and reconstruct them for signatures
#
cwebber
also hey
#
cwebber
everyone loves content addressed storage right?
#
eugr
okay here's a thing
#
cwebber
dlongley / dlehn: btw how do you store these references in your db, since I think you do that
#
cwebber
if you're storing your documents, how do you actually save which revision is being pointed to in json
#
eugr
i need to have some old fashioned models in my application because anything else requires me to code common functionality myself
#
eugr
because everything expects normal models
#
cwebber
eugr: you using something like mongo or rails?
#
cwebber
er
#
cwebber
django or rails
#
cwebber
or something else standard-framework like that?
#
eugr
like, my Video model has an uploader via Shrine
#
eugr
yes rails
#
eugr
if i encode that stuff in a weird way i'm not gonna be able to use any existing tooling
#
eugr
so, i imagined that i would build a local-only application first
#
eugr
in an old school wa
#
eugr
y
#
puckipedia
I use EntityFramework, the standard .NET ORM, but then basically hacked my own data storage on top. so I can easily poke something else (e.g. IPFS) over it
#
eugr
and then i would add AP on top
#
cwebber
eugr: if I were you I'd design for federation from the get-go... adding later is always harder
#
cwebber
eugr: but, it may not be in this case
#
cwebber
because at least you know what to expect
#
cwebber
a lot of the pain comes from learning about all the corner cases of federation after the fact
#
cwebber
of most of your structure being there
#
eugr
so hooking AP on top means, be able to serialize those models into AP, which i've successfully implemented for Actor and Video
#
eugr
it also means the fiddly bits, payloads
#
eugr
like Create, Delete, Add, Remove
#
puckipedia
you know. I really need to fix Kroeg's IDs
#
puckipedia
also I shouldn't give Links IDs
#
eugr
so my outlook so far is
#
eugr
when certain actions on my models are performed, i need to fire off the correct activitypub activity, and record the json, and send out the json to subscribers
#
eugr
make sense?
#
puckipedia
yep
#
eugr
so i thought my activities table would allow me to create an outbox by simply SELECT * FROM activities WHERE account = ?
#
eugr
so let's see, how would i change that to incorporate your suggestions
#
cwebber
eugr: mostly true, though you may develop some things dynamically
#
cwebber
eg, let's imagine you make a collection
#
cwebber
is that whole collection stored in your database as one object
#
puckipedia
eugr: I have another table to map a collection entity to its objects
#
cwebber
or is it, itself, dynamically generated partially
#
cwebber
eg imagine you and I are sharing a photos collection
#
cwebber
do you serialize all that pagination and everything into the json field?
#
cwebber
probably not, there's probably some dynamic additions
#
cwebber
*eugr is unhappy with me*
#
eugr
ughhh
#
eugr
let me think about this
#
cwebber
I'm just the messenger don't shoot me!
#
eugr
let's say i want to add a video to a playlist
#
eugr
it's an Add activity that references a Video and a Playlist
#
cwebber
yep
#
eugr
but it does not have to contain either except for ID
#
cwebber
right
#
cwebber
though, you do need to know effectively for signatures a tuple of
#
eugr
other than that, i don#t think any Collections are actually needed to be stored like that
#
cwebber
[id to object, revision it was signed with, was it "embedded"]
#
cwebber
eugr: but what about *the collection itself*
#
eugr
Outbox is dynamically generated
#
cwebber
inbox/outbox, obviously those are
#
eugr
if you access a Playlist, that would be a dynamic model
#
cwebber
my argument is that's also true for the playlist
#
cwebber
yes
tantek joined the channel
#
eugr
the Playlist would reference Video items which are also dynamic
#
cwebber
eugr: what I'd do is allow each type to provide a method to dynamically expand itself in ruby
#
eugr
wut
#
cwebber
ok ignore me :)
#
cwebber
well
#
cwebber
I'm just talking about how I'd structure it
#
cwebber
I think Collections are one clear case but
#
cwebber
I think there may be others where you may end up dynamically adding some things...
#
cwebber
but tbh I can't think of them off the top of my head
#
puckipedia
Person endpoints?
#
puckipedia
I dynamically generate those
#
eugr
one thing that bothers me even if i implement this all so far it's local-content-only
#
eugr
i would need to normalize remote videos and actors into the same dynamic tables
#
eugr
like on mastodon
#
eugr
i don't think there's a way around that
#
puckipedia
do you need all the info a normal ActivityPub actor has
#
cwebber
eugr: yes there's some more dynamic expansion stuff done on local things only
#
cwebber
but for remote things you present the object "as you got it"
#
eugr
ideally i would only re-reference remote objects by ID
#
eugr
never embedded
#
puckipedia
I have a flag to do so :P
#
cwebber
eugr: but what happens if you got an object that was signed tthat itself embedded an object :)
#
cwebber
then you need to reproduce the embedded object which was as-you-saw-it-when-you-got-it
#
cwebber
hence CAS
#
eugr
okay, how do i do CAS
#
puckipedia
eugr: you hash the objects as they come in, then you store those in one table, then have the metadata (ID, etc) in another table, pointing to a specific hash
#
eugr
hash how
#
eugr
sha1
#
eugr
md5
#
eugr
something else?
#
eugr
normalization before?
#
puckipedia
normalize before, of course. you can use the thing that json-ld signatures does
#
eugr
how will that actually help me if an object comes in referencing another by ID
#
eugr
i won't be able to know which hash to use
#
puckipedia
you only need the older ones to recreate signatures
#
eugr
yeah how would i recreate the signature?
#
puckipedia
by looking up the ID, which then returns a list of "it embedded this object, which had this hash, which embedded this object, ..."
#
eugr
wow ok i dont get this at all after all
#
cwebber
eugr: if they don't embed it then it doesn't affect what their content was
#
eugr
i'll do the thing without hashes first
#
cwebber
eugr: if you think of it in triples
#
cwebber
if you embed it, the flattened triples will include all the content if it's embedded
#
cwebber
but if you don't it will just say "it goes to this here..." but say nothing about the contents
#
cwebber
eugr: it's tricky! sorry for throwing so much at you but you asked what the right way to do it is
#
eugr
i dunno folks there's not that much to be done with signatures tbh
#
cwebber
well there's a lot more potential for signatures than just for forwarding objects but I have a feeling if I start talking about it I'm going to look like an even crazier person deep in teh weeds than I already do right now ;)
#
eugr
i am not building a general purpose server
#
eugr
like
#
eugr
this thing is gonna ignore everything things it can do
#
eugr
*except
#
eugr
like it's gonna understand Videos, and eventually playlists
#
eugr
it should ideally understand comments (replies to Videos)
#
eugr
it would send out non-Video activities so that e.g. mastodon would get notified about new uploads
#
eugr
but it's not gonna do microblogging
#
eugr
the home feed will only display video subscriptions
#
cwebber
ok :)
#
cwebber
well! you can ignore my recommendations
#
cwebber
you asked for thoughts on the right structure and I laid forth my tale-of-the-choices-I-wish-I-made ;)
#
cwebber
but "great is the enemy of good" so!
#
eugr
oh hey
#
eugr
what about the @context property
#
eugr
can children have it?
#
eugr
should i store it in the objects table
#
puckipedia
eugr: I would suggest expanding/compacting whatever object you get with your own context
#
puckipedia
and then you can just skip storing the context
#
Loqi
[ruby-rdf] json-ld: Ruby JSON-LD reader/writer for RDF.rb
#
puckipedia
actually, I think you can directly compact any input you get with your own context
#
puckipedia
so yeah, just that JSON-LD's compact, with e.g. a Mastodon-like @context
#
puckipedia
this way you can just use default json handling to read it, and not have to worry about someone thinking "eh, I'll use toot:manuallyApprovesFollowers instead of manuallyApprovesFollowers"
#
cwebber
yeah you can compact to a local context without needing to expand
#
cwebber
(expansion is automatically done for you as an intermeidate step where necessary)
#
eugr
so i'm gonna generate UUIDs for activity IDs, i guess
#
eugr
but they need to be URLs so how should i structure them
#
puckipedia
whatever you want
#
puckipedia
I would probably say /@:username/activity/:uuid, so you can have /@:username/video/:uuid etc
#
eugr
ok
#
Loqi
[puckipedia] #5534 Allow ActivityPub Note's tag and attachment to be single objects
#
puckipedia
took me a while to figure out why my DMs weren't arriving on Mastodon's side
#
cwebber
I'm doing some goofy experimentation trying ot get the websocket ping stuff working so test.ap.rocks is going up/down for a bit
#
puckipedia
I've worked around the Mastodon issue on Kroeg, but only on my puckipedia.com instance
#
cwebber
woo websocket pinging is working
#
cwebber
let's see how long https://test.activitypub.rocks/ stays alive now ;P
#
cwebber
I'm gonna go play some Cataclysm. Hopefully in 15mins it's still up :)