Ten years of sakamoto.pl: my thoughts and wishes

Ten years of sakamoto.pl: my thoughts and wishes

It's been ten years since I started self-hosting 24/7! Well, around 10 years - I don't have an exact date, and in-between, we cumulatively had a few weeks of downtime - but I think it still counts.

I didn't keep accurate records of everything that happened (and regardless, I don't wanna bore you to death); This post is a vague recollection of events, plus some thoughts that have been floating in my head and desperately want out.

Preface: server Mk notation

Throughout this blogpost I'll be referring to sakamoto with a "Mark" (Mk) next to the name, to refer to a specific hardware configuration. Mk is bumped every time a large hardware upgrade takes place, like a motherboard replacement. For your convenience, here's the (rough) version history:

Mk	Hardware	Time period
1	Raspberry Pi 1B	2016 - late 2016?
2	Raspberry Pi 1B+	2017 - 2018?
3	Raspberry Pi 3B	2018 - 2020
4.0	intel desktop board (Xeon X3360, 8GB DDR2, described here)	2020 - 2021
4.1	4th-gen intel core i5; board I rescued from having bent pins in the socket; it died a few months later of VRM issues	~2021.02 - 2021.04
4.0	surprise! out of options, we downgraded to the previous board, until THAT BROKE TOO	~2021.04 - 2021.07
4.2.0	chinese Xeon board, 16GB DDR3 ECC, later upgraded to 128GB because it was really cheap. Originally my desktop.	2021.07 - 2024.05
5.0	ASRockRack AM5, Ryzen 7590X3D, 192GB DDR5. Board was faulty and would randomly reboot under load.	2024.05 - 2025.03
5.1	Supermicro board, specs the same as 5.0, except it's not broken	2025.03 - present

First: but why?

I was 14. I've already been an avid Linux user for a couple years by then, and I enjoyed it lots. Also, ever since I remember I had some sort of an attachment to servers and networks - they felt cool, and I wanted to be a part of "it", whatever "it" really was.

Somewhen around that time I also lost a whole lot of my old data in a repartitioning accident - instead of resizing a partition, I deleted it, and I never managed to recover my data. I didn't trust my local storage afterwards, I really needed something external, with better backups.

Between all of this, there were also my parents, who - to put it very mildly - weren't very good at being parents. I'd get my access to internet limited in a variety of ways, sometimes for weeks at a time. I don't know what was their intent in having me constantly grounded, but it taught me MAC spoofing, and the patience of downloading stuff on a 4KB/s GSM connection, as that was the only internet access I could afford. At my lowest, I remember asking classmates for their WiFi passwords and "camping" in front of their apartments while I waited for my PSX games to download. "Fun" times.

As a culmination of all of those problems, I concluded that:

1. I need a central dumping ground for files downloaded off the web, for later access from my PC and phone, and...
2. I need a better place for data archival

Picture 1 - February 2017. sakamoto Mk2 was hooked up to a pair of trashpicked speakers (this picture is before I repaired the membranes, heh), and one of my spare screens. I used to watch youtube like this, with mpv over ssh.

Naturally, the outcome was the server. Initially called simply rpi, soon renamed to sakamoto, named after the cat. The original config consisted of a raspi 1B, a USB to SATA bridge, and some random SATA drives I managed to buy 2nd-hand. The intent was to create a NAS, and not much more. This retroactively became sakamoto Mk1.

Initially only available on the local network, it was soon expanded: first by becoming my intermediate router (creating a separate network just for devices in my room). Not much later I figured out SSH reverse-proxy, and started forwarding ports for my webpage, SSH, and a stash of files.

Picture 2 - May 2017. sakamoto Mk2(?) in the back, Hakase Mk1 in the front, next to a directional 2.4GHz antenna, because I was forced to use WiFi for uplink. I had a second WiFi card attached, which would transmit my own network with SSID "sakamoto"

I learnt a whole bunch about networks and general sysadmining. My parents never really recognized me for it - quite the opposite, they tried to shut the operation down, saying that it's pointless / stupid, and arguing how the raspi is needlessly wasting power. And heck, it's not like I wanted to use a raspi: i chose that because I expected them to whine about power usage!

Picture 3 - xkcd 705

For better or for worse, that environment shaped me into someone similar to the sysadmin from xkcd's Devotion to Duty. I remember one summer, when I came back a few days early from a summer camp, because sakamoto went down - only to find the main power cable disconnected from the wall, with everyone in the house avoiding responsibility...

I'll leave imagining how alienating that must have felt as an exercise for the reader. Picture 4 - June 2017. First factory-new drives I bought for the server. I got a really good deal on the purple, and it turned out to be nearly silent. The blue was for backups.

It's really funny looking at the creation date of the picture above. 9th of June, 2017. One day later I fucked up copying the files over, and got all the modification dates reset. Thus, 2017-06-10 became the de-facto epoch for most files on the server. I still have a bunch of stuff with that modification date.

---

Moving out for high school, I took the server with me. In my 2nd year, I saved up enough to do a large upgrade: I bought two almost-new 4TB WD Red drives, got a free motherboard from Remiberry and elly, and 16GB of DDR2 as a (birthday?) gift from Lili. That was 2020, and I actually wrote a blogpost about that upgrade! Word of caution: I've gotten a little bit better at expressing myself in text since then.

Picture 5 - sakamoto Mk4, February 2020.
someone tell me from 6 years ago to turn on some DAMN lights

That's when sakamoto went from Debian on ARM to Alpine on x86_64. Since then we've migrated servers a bunch of times, but the OS is still the same Alpine install. Six years later, almost no regrets; It breaks very rarely compared to anything else.

Picture 6 - at one point we had not one, but Two IRL servers: sakamoto for storage-intensive tasks, and Hakase for perf or memory-intensive tasks. That got reduced to sakamoto hosting virtually everything due to the raising energy prices.

Two house moves and one city later, we strategically relocated to get internet from bgp.wtf. They're the best ISP I've ever interacted with. We rented a range of public addresses (mostly so we could offer VMs for friends), and a 500/500Mbps connection, later upgraded to a symmetrical gigabit. The speed may not be particularly impressive today, but for Warsaw in 2021, that was the best one could get without shelling out for a colo.

Picture 7 - i still miss that room

And well.. about colocation.. We did "shell out" for it, for purely practical reasons (I no longer live in Warsaw). It's still hosted by bgp.wtf - not only is it amazingly cheap for $current_year, but having the infra around sakamoto managed by people I know and trust is really cool.

Picture 8 - October 2024, sakamoto Mk5.0, racked up.

Somewhere between this and that, we expanded what exactly we offered to the outside world: a few Linux mirrors, a git forge, and a couple other things. There's now a whole list on sakamoto.pl - check it out, and feel free to use anything useful!

okay okay skip to the chase

Selfhosting isn't for everyone. It's not an action that's done once and forgotten about - sure, some servers require little to no maintenance, and ideally you should design stuff to keep working for as long as possible without you around - but the mental price is still there, just hidden.

It eats at you every time something breaks, because you worry about uptime and user data. Every time someone notices that your service is down. Every time someone doesn't notice that it went down, too!

I think that's what ultimately differs me from "homelabbers": I like tinkering, but my system had a purpose in mind from the very start, not just "do this because it's fun". And nowadays you can't even call me a LARPer, as we have hundreds, hell thousands of people closely depending on the stuff we host.

It's okay for things to go down, sometimes

I see a lot of people in "devops" circles pushing for high availability, full redundancy, clustering, kubernetes, helm, whatever else is popular right now. Not just for "professional" infra, but also for hobbyists. If you're one of them, I want to make it absolutely clear that I'm not here to kill your fun - if this is something that excites you, please continue doing it!

That being said: all of those paradigms concern the last .1% to 1% of uptime. The part that's normally attributed to luck, or just shrugged off, because all infra needs maintenance. Optimizing that remaining 1% of uptime will cost you 10x as much time as the first 99%, and sometimes you will still lose the battle, because external factors cannot be controlled, and stuff may just break.

They say it's worth it, because "services need to stay up". I'm questioning whether the added complexity of hosting a static page on a geographically-distributed kubernetes cluster is really worth the added effort.

If it's not High Availability, clustering is "cool" because it lets you distribute the traffic, and manage the load!

... Except I've been doing none of this, and somehow my infra is still alive. My page (the one you're reading this on) is a PHP script I wrote back in high school, taking somewhere in the ballpark of 5-10ms to render. It doesn't have a database (it doesn't need one!), the files are largely static with very little actual code running, and the most processing it ever needs to do is reading a JSON file to render the comments into HTML. I could run this on a Pentium II, and it would STILL work reasonably well.

Perhaps my hottest take after doing this for 10 years and observing what others do (and fail to achieve), is that lowering the complexity is Good, Actually. Simpler systems == less needed maintenance == less mental health burden on the operator == easier to teach others how to help.

Oh! Others, helping!

How it became a collaborative project

"It just kinda happened."

sakamoto had user accounts for my friends essentially since the start. The only rule was (and still is) "don't shit on the carpet", and it seems to work fairly well, at least for the people I hang out with.

The server turned out to be an amazing outlet for folks who wanted to host stuff but don't have the means to do it (or don't want to host a server themselves). As a sysadmin, I'm doing the maintenance, they just... enjoy the free space for doing cool stuff. We've had everything from webpages, through file sharing, up to game servers, and it all spawned a small community of my friends interested in helping the cause. I'm very thankful for that!

At one point when looking at ls /home, I realized that I'm hosting a pubnix: sure, it's a pubnix for friends, which makes the community management much easier, but it's still a pubnix.

Picture 9 - A 'wall event', somewhat of a staple of our community. A bunch of people log onto the shell and broadcast messages onto the UNIX wall. Great fun! Picture 10 - This specific wall event was special.
shebang came over to warsaw, and logged onto the server on the local console.
Sadly, this is how we learnt that broadcasts don't properly show up on "physical" ttys.

One interesting (and unexpected) challenge turned out to be teaching others how stuff works - myself, I know where things are, older users know where things are as well, but the stack grew out so much that telling others "/mnt/pub is where the file stash is, have fun" is simply not sufficient anymore.

So I started writing a manual:

Picture 11 - nothing special, it's just basic HTML with links, and a custom helper that lists available topics, available under about

It's not much, but... it makes me happy that we came so far as to even have a need for something like this. I like technical writing, and being able to help my users feels really nice.

Oh, feelings!

Hosting is Awesome! ...but it also Sucks!

I love networking and servers. Having a place for my garbage (affectionate) is one of the coolest things in the world for me, and being able to share this with others (and give them as much as I can) is also really cool!

But recently, for every cool part about the server hosting experience, there's another part that Sucks to match.

• Like scrapers, who went from a minor nuisance that could be ignored, to a threat which required a whole custom system to mitigate.
• ...or script kiddies trying to DoS our services, rarely succeeding in bringing anything down, but always slowing down legitimate users.
• ...or spambots, trying to use confirmation emails from our MediaWiki instances to spam random addresses. Hell, not to mention normal mail spam that we receive!

Hosting servers is one of my passions - one of very few things that I can see myself doing forever without getting bored. The stakes have been progressively getting higher: first a few friends, then users of our Linux mirrors, later a mid-sized Fedi instance, servfail, and everything else in-between.
Sysop life stays the same, but when I look back, it's all so different...

At times, when things went out of my control, I've considered giving up. Throwing in the towel, saying "we had a good run, i'm shutting this down next month". The past 10 years haven't been an easy ride, and while it's still incredibly rewarding, I'm painfully aware that there will come a day when I'll have to make the hard decision: either hand sakamoto off to someone else, or shut it down. Not today, not tomorrow, but some day. I'm dreading the day when this happens.

One of my largest dreams is to turn this project into something I could genuinely hand off to someone else, so it continues in perpetuity when I'm no longer around. Whether that's feasible, I don't know - but I can dream.

so. was is it worth it?

Short answer: Yes.

Long answer: uhh...

Caring for sakamoto (and my infra as a whole) shaped me as a person. I wouldn't be who I am if not for this. Hell, I probably wouldn't be around anymore - at my lowest, the desire to keep my servers up was the last line of defense from getting myself hurt. (Laugh all you want, it worked!)

It's how I got my first job: after 3 months of sending offers daily, I got an interview, where I mentioned that I've been already hosting servers for years, and that I know what I'm doing.

I didn't really know what I was doing.

Nonetheless, my infra seemed to convince them enough to hire me, and all the subsequent recruitments were much easier.

Hosting stuff is also how I met lots of friends, and (hilariously) it's also a great conversation starter. If you've never invited someone you thought was "very cool, but scary to approach" onto your file stash, did you even live?

However, the mental health burden of it all cannot be ignored. I probably would be a healthier being if I didn't worry about gesticulates wildly everything! I'm already sleeping better ever since I delegated the startup procedure to a few trusted friends.

And yet, all of this is still manageable. The pros/cons sum is still positive. So we keep on truckin'!

incapable by nature of being seen : not perceptible by vision

Do you remember how I mentioned that hosting stuff is a long-term commitment? How it's not one-and-done, rather something you need to keep caring about long-term? Wouldn't it be ironic if upkeep was one of the most invisible jobs in tech?

Because it is. And it's really ironic.

Your users will only notice that you host things for them when those things stop working. Be prepared for this. If someone reaches and thanks you out of the blue, it's your lucky day! Adjust your expectations accordingly.

should you do it?

... is a question you should ultimately answer by yourself. If you want to do it for real, not as a hobby homelab running 4 hours a week "when you feel like it", then it will be a commitment.

My suggestions:

• Design the system to be as easy to manage as possible. Less friction for you == less of a chance that some day it'll be just too much to handle.
• Share stuff publicly! We don't need more datahoarders that never share stuff they found; If possible, copy that floppy - publicly, or just to friends.
• Document stuff.
  • Not necessarily in a central place (but that's fine as well)
  • a comment in a config file above something Weird is worth a thousand curses when you break it the next time.
  • Having little "changelogs" at the top of the file is always cool! You can see how long you've been cursing at this, and track your progress to make sure you're not going in circles.
• Universal tools are Awesome.
  • crontab will work just as well in 10 years as it did 30 years ago.
  • Unfortunately, same can't be said about most new tools. I've had much less migration burden by sticking to tools that aren't a moving target.
  • Before jumping wagon on the "new cool thing", consider if replicating the same thing with standard POSIX tools wouldn't be more future-proof
• Do things poorly if you can't do them well
  • Having something deployed at all is better than procrastinating it ad infinitum
  • Launching stuff on tmux instead of making a service is Okay, Actually. you probably should make that service at some point, but right now? Hecking go for it.

---

---

Comments: