I moved my Mastodon server into my house
DMV.community has lived on a Vultr VPS for a while now. This week I moved it onto the server that sits in my house and already runs a few other things I self-host. It's a Ryzen 7 3800X with 32 GB of RAM and a fast NVMe drive, which is a lot more machine than the VPS ever was, and I'm already paying for the electricity.
It went pretty well. It also went sideways in a few fun ways I didn't see coming.
The move itself
The actual cutover was the boring part, which is what you want. Stop Mastodon on the VPS, pg_dump the database, restore it at home, start everything up, flip DNS. The plan budgeted 20 to 35 minutes of downtime for all of it.
A few choices made that easy:
- Nothing is exposed from my house. Traffic comes in through a Cloudflare Tunnel, so there are no open ports on my router and nobody can see my home IP from the outside.
- Media stays in the cloud. Images and videos already lived in an S3 bucket behind a CDN, so none of that moved. My home upload speed never has to serve somebody's cat video to every server that follows us.
- Everything runs in rootless Podman containers, each one managed as a systemd user service. Postgres, Redis, Elasticsearch, the web server, the streaming server, and a handful of Sidekiq workers.
The search index had to be rebuilt from scratch on the new Elasticsearch, which took about an hour in the background. Search was a little dumb for that hour, and that was about it.
The security thing that surprised me
This was the part that took the most work before the move, and I'm glad I didn't skip it.
A Mastodon server spends all day fetching URLs that strangers hand it: posts from other servers, link previews, profile pictures, videos it has to process with ffmpeg. On a VPS, the worst case of something going wrong is that one VPS. In my house, the same server sits next to my smart home devices, and every device on my network.
And by default, a rootless Podman container can reach services on the host's localhost and the rest of the home LAN. That's a very bad default for software whose whole job is fetching arbitrary URLs.
So the stack got locked down pretty hard:
- The database, Redis, Elasticsearch and the streaming server sit on an internal-only network with no route to the internet, the LAN, or the host. They can only talk to each other.
- The web server and workers get a second network for internet access, plus a firewall inside Podman's network namespace that drops anything aimed at private address ranges, the host, or the LAN.
- Every container drops its Linux capabilities, runs with no-new-privileges, and has memory/CPU/process limits, so a runaway media job gets killed by its own container instead of eating the machine.
- New passwords for everything, Redis locked down. The old box had a couple of shortcuts I'm not proud of. They didn't make the trip.
Then it was slow
Day one, posting felt sluggish. Everything worked, but there was that little pause after you hit "Post" that makes you wonder if it went through.
The culprit was the disk, sort of. My home server uses btrfs, and the database was sharing that drive with everything else on the machine. Postgres is very particular about writing things to disk safely, and btrfs's copy-on-write behavior plus a bunch of neighbors all writing at once meant those safe writes would sometimes stall for seconds. A stalled write meant a stalled post.
The proper fix is giving the database its own drive with a plain old filesystem, so I ordered an SSD just for Postgres. Until it shows up, there's a temporary bandaid. I told Postgres it doesn't have to wait for every single commit to hit the disk before saying "done." The tradeoff is that a power cut could lose the last second or so of activity, which I can live with for a couple of weeks. I have the server plugged into a UPS
I also shut down my GoToSocial instance for now. It turned out to be the single biggest writer on that drive, more than Mastodon itself, so it's on ice until the new SSD is in.
The tune-up
Once it was running, I had my Hermes agent dig through the whole stack to see what else could be tuned for this hardware.
Some highlights:
The nightly cleanup job was killing the website. Mastodon needs regular cleanup of cached remote media, and that job was running inside the web server's container, sharing its memory limit. A long cleanup would use up the memory and the kernel would kill web server processes to make room. It now gets its own throwaway container with its own memory budget. Fun fact: the first cleanup run deleted 195,000 cached remote media files, around 346 GB worth, out of object storage. That's going to make a nice dent in the storage bill.
Every log line was being written twice. Podman was logging each container's output to the system journal, and systemd was also logging it, so every line showed up twice. Combined with the background workers logging every single job they ran, that was hundreds of thousands of lines an hour. The journal hit its 4 GB cap and was rotating out old logs early. Now each line gets written once, and the workers only log warnings and errors.
One database query was doing way too much work. The most expensive query on the whole server checked whether any account from a given server had a particular setting turned on. For big servers with no matches, Postgres had to look at every single account from that domain to answer "nope." One bridge server alone has over 150,000 accounts in my database. A small 7.6 MB index took that query from 352 ms down to about 8 ms.
Redis was allowed to throw away jobs. Redis holds both Mastodon's cache and its background job queue. It was set to evict any key when it got full, which in theory includes jobs. I switched it to only evict keys with an expiration time, which is all the cache stuff and none of the job stuff.
Push notifications broke for some people. About 30 users get notifications through ntfy.sh, and after the move ntfy.sh just... stopped answering my home IP. Everything else worked fine. My best guess is that a burst of backed-up notifications from a brand-new IP set off an abuse filter. The fix was a little relay: notifications headed for ntfy.sh now hop through another server I run in a datacenter. It's just passing encrypted traffic along, so it never sees the contents. Notifications started flowing again within minutes.
A few smaller things got fixed too: Postgres settings that assumed more RAM and CPU than its container actually gets, and a monthly cleanup job with a SQL bug that would have quietly done nothing.
What's left
- The SSD. When it arrives, the database moves to its own drive and the temporary bandaid comes off.
- A big cleanup. About half of the 24 GB database is leftover link-preview data for posts that no longer exist. Mastodon's own cleanup deletes old posts in bulk and leaves the link-preview rows behind. Clearing that out before the SSD move means the database lands on the new drive half the size.
- GoToSocial comes back once the database has its own drive.
Was it worth it?
So far, yeah. The server has way more headroom than it did on the VPS, and I understand how the whole thing works even better than I did a week ago. Nothing like a server migration as a learning opportunity. Also, monthly hosting costs will drop significantly. The only thing left to pay for in the cloud is S3 and the media proxy. The trade is that it's in my house now. If the power goes out for a long time, DMV.community goes out with it. The status page lives on a VPS though, so it will always be accurate.
If you're thinking about bringing a Fediverse server home with you, my two cents: put the database on its own disk from day one, and assume your container runtime's network defaults are friendlier than you want them to be.