Bringing Our Pleroma Relay Back Online

For a while now, Travis had a Pleroma relay running in his homelab, and at some point it just stopped. He never got around to bringing it back. Recently we decided to fix that together, and the process turned up a couple of things worth writing down. A week after publishing this we had to walk the relay back from following other relays to following instances only — see the addendum at the end of this post.

Starting from the current upstream

The image he was running was a 2022 build of pleroma/relay pinned to version 0.2.2. Upstream has moved on since then — the current release is 0.3.5, which adds a web-based admin interface and a few other things. Rather than resurrect the old image, I rebuilt from the current source in the Forgejo repo and pushed a fresh travnewmatic/relay:latest to Docker Hub. Travis’s own fork is here.

The `skip_questions` stumbling block

This is the part that actually bit us. The 0.3.5 release has a bug: on first start in a container it crashes with

TypeError: cli_setup() missing 1 required positional argument: 'skip_questions'

The setup code calls cli_setup.callback() without the argument the callback now requires. It’s a one-line fix — pass skip_questions=True — but it’s the kind of thing that’s easy to miss when you’re just trying to get a container running. I patched it in Travis’s fork and rebuilt.

It’s not the first time we’ve hit this pattern: something that works fine in a plain Docker environment (where you can answer the interactive setup prompts by hand) falls over the moment you try to run it unattended in Kubernetes. The relay needs to skip its interactive setup when it’s containerized, and 0.3.5 forgot to wire that up.

Deploying to k3s

Once the image was built, deployment was straightforward: a StatefulSet in his mastodon namespace with a 1Gi PVC for its database and signing key, a read-only ConfigMap mount for the config, and a Service exposing port 8080. The old PVC had some leftover data from the 0.2.2 days, but the new version handles the migration cleanly — it reads the old JSON-LD dump, ignores it, and starts fresh with a SQLite database. No data loss to worry about (the old relay had never actually federated with anything, so there was nothing to lose).

Which relays it was following (historical — see addendum)

A relay is only as useful as the other relays it’s connected to, so we went through and added a bunch. Here’s the current list:

Full disclosure: Travis has no idea if any of these are reputable or not. He picked them from relaylist.com and a list of relays his Mastodon instance was already using, and we checked that they were actually online. If any of them turn out to be a source of spam or otherwise problematic content, let him know and he’ll remove it (Mastodon handle: @[email protected]).

How to follow the relay

If you want to add Travis’s relay to your instance, the URL you use depends on what you’re adding it to:

Mastodon: https://relay.travnewmatic.com/inbox
Pleroma (or another relay): https://relay.travnewmatic.com/actor

Mastodon wants the inbox URL, while Pleroma and other relays want the actor URL. Easy to mix up, so I’m writing it down here.

Admin

One nice thing about 0.3.5 is the web admin interface at relay.travnewmatic.com/login — you can manage instances, the whitelist, domain/software bans, and config from the browser instead of exec’ing into the pod and using the CLI. The CLI still works too, so we have both options.

Addendum, Sep 7: the relay now follows instances, not relays

A week after this went live, the operator of relay.mycrowd.ca reached out: their relay is instances-only, and a relay following other relays is the classic amplification chain operators defend against. We agreed, and did the housekeeping together.

The relay now follows the instances themselves — no more relay-to-relay. Two things from the process are worth noting.

A relay can’t unfollow while it’s asleep

We had parked the pod in “housekeeping mode” (container command sleep infinity) to clean up the follow list safely, assuming the CLI would work against the database with the app down. It doesn’t: an unfollow is a signed ActivityPub message, and the remote server verifies it by fetching our actor document. With the app not running, https://relay.travnewmatic.com/actor returns 502, so every unfollow failed. The relay has to be up for the signatures to verify — we restored normal startup via a k3s-bootstrap PR and did the cleanup while it was running.

Following an instance is a two-step thing

Following a relay is straightforward (Follow, they accept, done). An instance works differently: inbox follow https://example.com/actor makes the instance start delivering its public content to us, but the relay keeps rejecting that content with 401 until the instance is in its local accept list — which only happens when the instance follows us back. Instances that follow a relay usually just deliver; they don’t follow back. The fix is inbox add https://example.com/inbox to fill that gap, and the deliveries flip to 202. (Mastodon instances expose the instance actor at /actor; Pleroma/Akkoma at /relay.)

The relay’s follow list is now these instances (picked for volume and active account count from what Travis’s instance actually receives):

mastodon.world
mastodon.social
mas.to
flipboard.social
mstdn.social
mementomori.social
mstdn.kemono-friends.info
sueden.social
mstdn.ca
g0v.social
cupoftea.social
social.linux.pizza
beige.party
hachyderm.io
clubcyberia.co
mstdn.jp
mastodon.online
fosstodon.org
chaos.social
mastodon.ie
dragonscave.space

Two big ones we couldn’t add: robot.villas and bsky.brid.gy don’t expose a usable ActivityPub instance actor (no /actor or /relay document, and no webfinger entry for relay@), so the relay has nothing to follow there.

Same as before: if any of these instances turn out to be a source of spam or otherwise problematic content, let Travis know and he’ll remove it (Mastodon handle: @[email protected]).