social system notes

July 19, 2021

This post contains my running notes on building the social-media features in a new plugin. It’s an experiment; I’m going to publish it publicly because there’s no reason not to, but its primary purpose is as a scratch-pad for my notes. So it will likely change a lot, with content constantly added, amended, and removed, and it may not make a lot of narrative sense. Feel free to ask questions if something catches your eye.

07/19 -

Started work on the server design. Added options for omitting the cookie-to-auth-header function from the access-control functions, since its only purpose relates to getting browsers access to stuff that won’t be included in this server. Also added an option for non-access-controlled paths in otherwise access-controlled sites, which is necessary for publishing public keys.

Been kinda dreading choosing a public-key format for this—heard a lot of shade thrown at the JOSE suite, which seems to be the most widely deployed. I was very happy to find this article which matches my intuition in the areas I know pretty well, and provides some well-supported thoughts stretching into the areas that I don’t. I will be using JOSE. Specifically, the JWK format for publishing keys seems well-supported and not especially controversial, so I’m feeling happy with it.

Basic plan:

An authoring UI based on the blog-post-authoring one. The main architectural difference is that social-media posts usually don’t have titles, and the user is not expected to supply an ID. This is going to affect both of the existing page designs; without a human-readable post-id, there’s no obvious feature for the list page as it exists. The edit page could be made the same-but-without-a-title-field. But it may make sense to combine the edit and post-list into a single feed page. Questions about that idea:

The function to process the uploaded posts is only a little different than the blog-post one. Instead of dropping posts in a blog bucket, it’s going to create a gzip archive of the main post md and all its imgs. This will be placed in a “posts” directory in the site bucket. The function will also add the post metadata to a dynamo table. The dynamo table will be the index of posts ordered by date—this is what will record what’s new on our site for when friends ask.

The site bucket will be exposed on a cloudfront distribution. The access-control functions on the distribution will receive requests from friends’ sites (signed with their published keys). It will verify the signatures against the published key, and make sure the signer is one of our friends, then passthrough the request to the endpoint.

There will also be a post-list endpoint on the cloudfront. This will be access-controlled by the same function described above, but instead of passing through to an s3 origin it will pass through to a lambda that queries the post-index table. It will return a list of posts with metadata—exactly which metadata is tbd.

Reader thoughts:

TODOS:

Rework post entry function from blog into a function for packaging social media posts as described above. Repurpose the existing post table to be the index table for metadata.

Write an access-control function (and I’m pretty sure that it is only one, as opposed to the 5 that were required for oauth) to validate incoming requests after fetching the jwks. Should we cache / pin JWKs from friends’ sites? What if any jwk rotation strategy makes the most sense. Like if a jwk unexpectedly changes and that’s something we want to care about, that has to include some kind of user interaction to see if the new jwk is trusted. But if 100% of people are going to click through that without reading it, then what would the point be. It might be good to just cache the known jwks and if they change make a new request (and silently use the currently-published one, whichever it happens to be).

Write a script to generate a jwk and store it. For now assume option 1: generate in a null resource in terraform.

Write a polling function to check for friends’ posts. This can maybe be a bit aggressive—every 5-10 minutes or so—since I have so far failed to make a meaningful dent in the lambda free tier allocation (of course, the scaling concern here would be more on the incoming side, where a multitude of requests from connected sites could incur lots of requests. We can cache the response from the index table though). Note that the polling function will be a POC for signing requests with the corresponding private key of our published key. Should the polling function also fetch posts, or should that be delegated? Implicates capacity questions (time limit, mem limit) that may want more specific answers than one-size-fits-all. So on balance the signing stuff should be modular in case we want to split up later.

Rework the authoring UI for social-post-authoring, answering the questions above. Use this to guide the friends’-posts-durability / storage question.

07/20

Copied the blog plugin to start off the social plugin. It seems like the only thing I might need to change on the server side is to have the post_entry (renamed to feed_entry) function replace its publish & img-publish steps with publishing a zipped archive. Steps:

  1. locally download a post & its imgs

  2. write a utility to parse the post, replace the img links with relative paths, archive all the files, return s3-ready buffer. Test w/ local post, imgs

  3. update or insert a step to get the available imgs (possibly filtered down to the used ones)

  4. insert the packaging utility as a generic async function step.

  5. update the published, pinned post metadata as described below.

Next, I’ll need to add a lambda endpoint to query the posts table & get the most recent published posts. There may not be a reliable “published” flag in that table. Idea for that:

  1. publishedPost is a new partitionkey value. Add a new index on a new createDate field.

  2. pinnedPost is a new partitionKey value.

  3. When an authed friend queries for posts, the lambda gives them the most recent 50 publishedPost (metadata only) and all of the pinnedPost metadata. The metadata includes urls to find the post packages. This response can be cached by cloudfront since every friend can see the same post list (if you want circles, deploy more than one social site).

UI changes

The index page turns into a feed, but retains a “new post” button that simply takes you to the authoring page. The feed shows you the latest chronological posts, yours & friends. Open question: what’s the strategy for gathering posts? If the requests happen just when you log on, that a) leaks information about your habits b) big latency hit on the ui. otoh, if you query friends every 10 mins, you only need to get the (likely cached) dynamo query results to know if you need to get any post packages. The post packages can be stored in s3 with an expiration time of a week or so (if you have a lot of friends / follows, this could become a significant amount of data if you kept it for long), potentially even rendered to html (by rendering md & being able to inspect imgs, as opposed to letting the distributor do the rendering, we have a bit of security). The holding location for the posts could be explicit-allowed as an iframe location, further isolating the rendered posts from the admin stuff.

The authoring page doesn’t change much, except that there’s no title.

Access Control

Have a null resource generate a jwk file (I think there might be instructions in the access control functions for doing it manually) and store the private key in the admin site s3 (hosted-assets? No, shouldn’t be accessible from UI. Uploads? No, shouldn’t be overwritable from UI. Some new path. Secrets? Config?). Store the pubkey in the public bucket. Permissions to read privkey is an array of role arns (at least the friend-query function, possibly a post-resolver if separate).

Look at indieauth again to see if it’s applicable here—it seems to have put thought into how to treat origin urls. otoh, since we know who our friends are I’m not sure this is critical for this use case.

Friend data should be kept in a dynamo table; the same table should hold the cached pubkeys. When a request comes in, its keyId is compared to the cached. If not match (or if it’s been a while since we checked) check to see if the site has rotated it’s key; use the latest published to verify the request. If verify, passthrough, else 401.

Poll

Write friend-poll fn. Try: get privkey, sign requests. every 10 minutes, poll all friends. If new, fetch, parse, store in post-storage area. Set 1week expiration (but what if pinned?)

07/21 08:12

In last night’s description I forgot the friend request / response flow.

UI: another page in the social plugin—friend management. Can likely be based on the current index page, which does table-based create update delete (but for posts). A “send request“ feature in the UI—validates, for a given url, that it is an instance of something that can respond to the request (tbh this is a pretty indieweb situation, hcard, the decisions made around indieauth seem relevant here and I remember them being sensible).

Backend: When the UI says “send a friend request to x” the backend does the same sanity check on the url (hcard, w/e) and through that sanity check discovers the location of the friend-request endpoint. It then constructs a friend request, details tbd, and signs it with the priv key associated with a published pubkey (use multiple keys, maybe, so we can distinguish which functions have access to which privkey).

In the friends table, friendships have a marker showing what state they’re in: friend request sent, friend request received, request mutually accepted. Friendships in the mutually accepted state are friendships for the purpose of access control. I’m not sure there needs to be a “request denied” state—at that point the record should be deleted (there might be some advantage to having denied as “reminder not to resend” but that energy would likely be better spent on filtering incoming requests from people one has previously denied, rather than flagging outgoing requests—which, maybe there should be a special “incoming-request-denied” state, so we can look skeptically at repeated requests from previously-denied sources.

The management UI has a filter & top bar showing counts by friendship state. Each action relevant to the current state is shown in the list.

That means that you can delete existing friendships. As a courtesy, a deletion should result in a notification to the other party (signed with the privkey) however we should not rely on that so we should make another friendship state of “disconnected” in case we get defriended by someone who didn’t send a notification. disconnected means that we didn’t get a notification but we are no longer able to read from the source—possibly differend disconnection states for network (there doesn’t seem to be anything there anymore) and auth (suddenly our published key is no good there, suggesting deliberate). A friendship in the disconnected state is not allowed for access control, but it can repair itself automatically to its previous state (beware of escalations here) if the disconnection passes. Repairing a disconnected friendship should be a monitored and measured thing, since it’s an opportunity for shenanigans.

So if a delete notification is received, the record is deleted or soft-deleted. It seems like in most cases soft-delete is the way to go, since it’s nice to have context when responding to incoming stuff.

07/22

Didn’t have any new realizations that I’d forgotten anything last night. Yesterday I did that bit of planning and added a social table but that was about it for this project. Today I hope to get the post-publish step to deposit a zip of the post contents in the public bucket.

07/24

Last night I got the post-packaging working. Today I want to go over the total post-entry function and make sure it all still makes sense. I think I have two good options about what to do next. I could start working on the access-control stuff, or I could work on the reader display of the packaged posts. I can’t think of a reason to prefer one over the other; I don’t see any obvious dependencies. I feel like I’d rather start with the access control. And that could easily turn into finishing the whole friend request flow before moving to the reader piece. That’s pretty on brand for me—first work on the plumbing and leave presentation for last.

If I want to start on the access control, what does that entail? First would be the null resource key generator. Answer the following questions:

  1. What are the details of the key—algorithm, alg params—use something common for a signing keypair

  2. Where exactly is the private part of the key stored? Is there already a path in the plugin structure where the plugin can store things intended to be secret from both the UI and other plugins, or do I need to make a new place?

  3. Where exactly is the public part of the key stored on the website? It should probably be under the `.well-known` path, but where exactly.