HomeLab #2: Immutable GitOps homelab + MikroTik hardening

#Before we begin…

First of all, huge thanks to the Hacker News community for all the feedback Part 1 of this series received.

Before diving into the homelab setup, I want to address one comment from that thread:

This is a good catch. Let’s close that gap first, before getting on with the actual homelab.

#Split WAN and LAN into interface lists

RouterOS firewall rules read cleaner against named lists than against raw interface names, and the same list works for both the Ethernet WAN port and the PPPoE client on top of it.

/interface list add name=WAN
/interface list add name=LAN
/interface list member add list=WAN interface=pppoe-out1
/interface list member add list=WAN interface=ether1
/interface list member add list=LAN interface=bridge

#Drop management traffic at the input chain

The default RouterOS input chain accepts almost anything bound for the router itself. Replace it with an explicit allow-list: established and related traffic, then LAN, then nothing else.

/ip firewall filter
add chain=input action=accept connection-state=established,related
add chain=input action=drop connection-state=invalid
add chain=input action=accept in-interface-list=LAN
add chain=input action=accept protocol=icmp
add chain=input action=drop in-interface-list=WAN

Order matters here. The drop in-interface-list=WAN rule sits last so it only catches what the earlier accepts didn’t already let through, and it catches everything, not just Winbox or SSH.

#Turn off services you don’t use, restrict the ones you do

/ip service binds Winbox, the web interface, the API, FTP and Telnet to every interface by default. Most homelabs need at most one or two of these, and none of them need to be reachable from the WAN once the firewall rule above exists anyway. Defense in depth means doing both.

/ip service disable telnet,ftp,www,api,api-ssl
/ip service set winbox address=192.168.88.0/24
/ip service set ssh address=192.168.88.0/24

#Stop announcing yourself

MikroTik’s neighbor discovery protocol (MNDP) is how WinBox found the router by MAC address during initial setup, back in part 1. It has no business running on the WAN side once the router is configured.

/ip neighbor discovery-settings set discover-interface-list=LAN

#Don’t forget IPv6

A private IPv4 WAN address behind CGNAT gets an incidental assist from NAT: unsolicited inbound packets have no session to match, so they never reach the LAN. That’s a side effect of translation, not a firewall rule, and it says nothing about IPv6. If the ISP hands out a routable IPv6 prefix, as most dual-stack IPoE and even some DS-Lite setups do, every LAN device gets a globally reachable address, and RouterOS’s default IPv6 firewall is far more permissive than the IPv4 one. Build the same WAN/LAN split for /ipv6 firewall filter, and don’t assume CGNAT is doing IPv6’s job for you.

/ipv6 firewall filter
add chain=input action=accept connection-state=established,related
add chain=input action=drop connection-state=invalid
add chain=input action=accept in-interface-list=LAN
add chain=input action=accept protocol=icmpv6
add chain=input action=drop in-interface-list=WAN

With the perimeter closed, the router is ready to carry something more interesting than home traffic.

Now, time for the homelab setup.

#The hardware: a gaming laptop

The cluster runs on a Lenovo IdeaPad Y700-15ISK, a 2016 gaming laptop that would otherwise sit in a drawer.

#The software: immutable server Linux distro

My OS of choice is uCore - a Fedora-based, container-native Linux image maintained by the Universal Blue project, built for running container workloads directly on bare metal. It layers hardware drivers and container tooling on top of Fedora’s rpm-ostree base, so the same rebase mechanism that ships the OS can also ship network drivers or anything else baked into the image.

“Immutable” here doesn’t mean the filesystem never changes; it means the OS ships as one versioned, read-only tree instead of thousands of independently upgradeable packages. rpm-ostree treats /usr and the default /etc as a single atomic OSTree deployment: an update swaps the whole tree in one transaction, and a failed update rolls back to the previous deployment instead of leaving the system half-patched. /var and /home sit outside that tree and stay writable and persistent across every rebase, which is exactly why /var/lib/rancher on the second disk survives an OS update untouched even though /usr gets replaced wholesale.

uCore has no ISO of its own. It’s an OCI container image built on bootc/rpm-ostree¹, and the documented install path is: install plain Fedora CoreOS (FCOS) with an Ignition² config that rebases the node onto ghcr.io/ublue-os/ucore:stable on first boot. Two rebases happen automatically, unsigned first and then signature-verified, each followed by a reboot.

#Flashing the installer and wiping both disks

Grab the current stable x86_64 metal ISO. Fedora’s build stream JSON resolves to whatever the current release is, so scripting the lookup beats clicking through the download page:

curl -s https://builds.coreos.fedoraproject.org/streams/stable.json \
  | jq -r '.architectures.x86_64.artifacts.metal.formats.iso.disk.location'

Flash it with balenaEtcher: “Flash from URL”, paste the URL given by the terminal, pick the USB drive, and press “Flash!”.

No dd incantation, and Etcher verifies the write afterward.

Boot the USB in UEFI mode with Secure Boot off, per uCore’s own install guidance: some of its bundled drivers ship signed with Universal Blue’s own key rather than Fedora’s, so Secure Boot needs that key enrolled as a MOK before it’ll trust them. At the live prompt, confirm which physical disk is which before touching anything:

lsblk -o NAME,SIZE,MODEL,SERIAL,TRAN
# both drives report TRAN=sata; match MODEL/SERIAL against the labels on the drives themselves

The plan for partitioning is as follows: M.2 SATA drive holds the OS, while the standard SATA SSD holds everything k3s writes. Splitting the two matters for a reason: uCore rebases overwrite /usr on every update but leave /var untouched, so putting workload data at /var/lib/rancher on the second disk means a bad OS update can break the host without touching a single PVC.

Warning: wipe both disks before partitioning.

coreos-installer install re-images its target disk destructively on its own, but that only covers /dev/sda. /dev/sdb’s partition table is created later by Ignition, from whatever state the disk was already in, so stale GPT/MBR headers, old filesystem signatures, or leftover LVM/RAID metadata on either drive can confuse the installer or Ignition.

Furthermore, /dev/sdX assignment order on Linux isn’t guaranteed across reboots or firmware changes, so don’t trust this mapping on your own hardware.

Before wiping anything, list /dev/disk/by-id/ via ls -la at the live-ISO shell to re-check, which /dev/sdX matches given drives:

core@localhost:~$ ls -la /dev/disk/by-id
(...)
lrwxrwxrwx.  1 root root   9 Jul 25 08:18 ata-Samsung_SSD_850_EVO_500GB_XXXXXXXXXXXXXXXX -> ../../sdb
lrwxrwxrwx.  1 root root   9 Jul 25 08:18 ata-Samsung_SSD_850_EVO_M.2_500GB_XXXXXXXXXXXXXXX -> ../../sda
(...)

Wipe both first:

sudo wipefs -a /dev/sda /dev/sdb
sudo sgdisk --zap-all /dev/sda /dev/sdb

#The Butane config

Butane³ compiles down to the Ignition JSON the installer consumes. This one creates the core user, partitions /dev/sdb as k3s-data, mounts it at /var/lib/rancher before k3s starts, and defines the two autorebase units as one-shot systemd services guarded by sentinel files so each stage runs exactly once.

cat > install.butane <<'EOF'
variant: fcos
version: 1.7.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAA... your-key-here
      password_hash: "$y$j9T$..." # generate with: mkpasswd --method=yescrypt
      groups: [sudo, wheel]
storage:
  directories:
    - path: /etc/ucore-autorebase
      mode: 0754
  disks:
    - device: /dev/disk/by-id/REPLACE_WITH_YOUR_DISK
      wipe_table: true
      partitions:
        - label: k3s-data
          number: 1
          size_mib: 0
  filesystems:
    - device: /dev/disk/by-partlabel/k3s-data
      format: xfs
      label: k3s-data
      wipe_filesystem: true
      with_mount_unit: true
      path: /var/lib/rancher
systemd:
  units:
    - name: ucore-unsigned-autorebase.service
      enabled: true
      contents: |
        [Unit]
        Description=uCore autorebase to unsigned OCI and reboot
        ConditionPathExists=!/etc/ucore-autorebase/unverified
        ConditionPathExists=!/etc/ucore-autorebase/signed
        After=network-online.target
        Wants=network-online.target
        [Service]
        Type=oneshot
        StandardOutput=journal+console
        ExecStart=/usr/bin/rpm-ostree rebase --bypass-driver ostree-unverified-registry:ghcr.io/ublue-os/ucore:stable
        ExecStart=/usr/bin/touch /etc/ucore-autorebase/unverified
        ExecStart=/usr/bin/systemctl disable ucore-unsigned-autorebase.service
        ExecStart=/usr/bin/systemctl reboot
        [Install]
        WantedBy=multi-user.target
    - name: ucore-signed-autorebase.service
      enabled: true
      contents: |
        [Unit]
        Description=uCore autorebase to signed OCI and reboot
        ConditionPathExists=/etc/ucore-autorebase/unverified
        ConditionPathExists=!/etc/ucore-autorebase/signed
        After=network-online.target
        Wants=network-online.target
        [Service]
        Type=oneshot
        StandardOutput=journal+console
        ExecStart=/usr/bin/rpm-ostree rebase --bypass-driver ostree-image-signed:docker://ghcr.io/ublue-os/ucore:stable
        ExecStart=/usr/bin/touch /etc/ucore-autorebase/signed
        ExecStart=/usr/bin/systemctl disable ucore-signed-autorebase.service
        ExecStart=/usr/bin/systemctl reboot
        [Install]
        WantedBy=multi-user.target
EOF

docker run --rm -i quay.io/coreos/butane:release --pretty --strict < install.butane > install.ign

#Installing and the double reboot

install.ign needs to reach the live ISO shell over the network; serve it from the directory where you ran butane:

python3 -m http.server 9000

From the live ISO shell on the target machine, point coreos-installer at that address instead of a local file:

sudo coreos-installer install /dev/sda --ignition-url http://<workstation-IP>:9000/install.ign

Once it finishes, pull the USB and reboot. UEFI lands on stock FCOS’s GRUB2 bootloader; if it doesn’t boot on its own, go into firmware settings and put the M.2 drive’s boot entry first. The 2.5” SSD has no EFI System Partition, so it can never boot on its own.

First boot runs Ignition (creates the user, wipes and mounts /dev/sdb), then the two autorebase units fire in sequence, unsigned then signed, each ending in a reboot.

Once it settles, SSH in as core and confirm the rebase actually landed:

cat /etc/os-release   # NAME should read "uCore"

rpm-ostree status     # booted deployment should point at ghcr.io/ublue-os/ucore:stable

#Locking down the laptop

Servers don’t sleep, but laptops default to doing exactly that when the lid closes. Disable it, mask suspend entirely as a backstop.

sudo mkdir -p /etc/systemd/logind.conf.d
cat <<'EOF' | sudo tee /etc/systemd/logind.conf.d/lid.conf
[Login]
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore
EOF
sudo systemctl restart systemd-logind
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target

Cockpit’s web login needs a password. If you added password_hash in the Butane config above (generated with mkpasswd --method=yescrypt), you’re all set. Install and enable:

sudo systemctl enable --now cockpit.socket
# dashboard at https://<node-ip>:9090

If you skipped password_hash in Butane, set one now with sudo passwd core before enabling Cockpit.

#k3s on the second SSD

The mount unit from the Butane file already puts /var/lib/rancher on the SATA SSD, and k3s stores everything there by default, so workload data lands on the right disk without any extra configuration.

curl -sfL https://get.k3s.io | sudo sh -s - \
  --write-kubeconfig-mode 644 \
  --disable=traefik

--write-kubeconfig-mode 644 lets your own user read the kubeconfig without sudo. --disable=traefik skips the bundled ingress controller because Flux manages ingress here instead. Confirm the node is up and the data volume landed where expected:

df -h /var/lib/rancher
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sdb1       466G   11G  456G   3% /var/lib/rancher

sudo kubectl get nodes
# NAME                    STATUS   ROLES           AGE   VERSION
# localhost.localdomain   Ready    control-plane   12h   v1.36.2+k3s1

Copy the kubeconfig into your own user so kubectl works without sudo:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown $(id -u):$(id -g) ~/.kube/config
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
source ~/.bashrc
kubectl get nodes

#Bootstrapping FluxCD

A single-node cluster still benefits from GitOps, if only so that “what’s running” always matches what’s committed. Install the CLI and sanity-check the cluster before touching anything:

brew install fluxcd/tap/flux
flux --version
flux check --pre

#The repository

Flux needs a Git repo to reconcile against. Create a private one, home-cluster, and lay it out with a root that Flux bootstraps into and an apps/ tree Flux watches separately:

home-cluster/
├── cluster/
│   ├── base/
│   │   ├── flux-system/          # populated by bootstrap
│   │   ├── apps.yaml             # Kustomization watching ./apps
│   │   └── kustomization.yaml    # root: apps + flux-system
│   └── apps/
│       └── kustomization.yaml
└── README.md

#Bootstrap

Generate a fine-grained GitHub PAT scoped to home-cluster with Administration: Read-write, Contents: Read-write, and Metadata: Read-only (a classic PAT with repo scope works too, just broader). Export it for the current shell only, then run bootstrap:

export GITHUB_TOKEN=ghp_XXXXXXXXXXXXXXXXXXXX

flux bootstrap github \
  --owner=<your-github-user> \
  --repository=home-cluster \
  --branch=main \
  --path=cluster/base \
  --personal \
  --token-auth

Bootstrap creates a deploy key, pushes Flux’s own manifests into cluster/base/flux-system/, applies them, and leaves a flux-system Kustomization watching cluster/base/ from then on. Confirm it’s healthy:

flux check
flux get all
# gitrepository/flux-system     True    stored artifact for revision 'main@sha1:...'
# kustomization/flux-system     True    Applied revision: main@sha1:...

Bootstrap only wires up cluster/base/; it doesn’t know about cluster/apps/ yet. Add a Kustomization pointing at it and a root that ties both together:

cat > cluster/base/apps.yaml <<'EOF'
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m0s
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps
  prune: true
  wait: true
  timeout: 5m0s
EOF

mkdir -p cluster/apps

cat > cluster/apps/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: []
EOF

cat > cluster/base/kustomization.yaml <<'EOF'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - apps.yaml
  - flux-system/
EOF

git add -A && git commit -m "feat: wire apps root" && git push

Flux reconciles within a minute, or force it with flux reconcile kustomization flux-system --with-source.

#Maintenance and a few sharp edges

A handful of things about this setup will bite if you don’t know them going in. Flux’s prune: true on the apps Kustomization means renaming or moving a tracked resource in Git deletes the old one and creates the new one at the reconcile that follows; be deliberate about mv, because Flux won’t ask twice.

Cluster live. apps/ stays empty for now, ready for whatever gets deployed onto it.