I'm always bothered by single points of failure. For my homelab, this quickly became a single HAProxy virtual machine. It existed on only one of my Proxmox instances (I can't replicate between them because the original instance doesn't use zfs. oops) so if the virtual machine or physical machine was unreachable for whatever reason, it effectively blocked my access to any service in my homelab Kubernetes cluster, and also broke communication between nodes and the Kubernetes API. It didn't happen often, and the fix was often trivial, but it still bothered me. Coupled with the fact that until recently the Kubernetes cluster also acted as a reverse proxy to services hosted on Proxmox and my NAS (read more about that stuff here), it could cause a wider outage than just what's on the cluster.
I was aware of things like keepalived and "virtual IPs" (VIPs), but procrastinated properly implementing them on redundant HAProxy instances. Part of the blocker was that Tailscale doesn't have any good way of doing VIPs - so while I could have a VIP for my home subnet, it wouldn't help when routing through Tailscale unless I opted for a subnet router. And then, that subnet router might not be highly available (I'm not sure how Tailscale subnet routers behave with overlapping subnets).
MetalLB was also on my radar, but I was unsure how it would work with my consumer grade router. While building a small business/enterprisey router is on my long, long list of things to do, it's a ways off so I didn't consider MetalLB until recently. During a rather boring day, however, I decided to poke around and actually deploy it to see if it would work.
Much to my shock, putting up a deployment in layer 2 mode just worked. I'm still very much in the learning phase of "advanced networking", but the fact I could just drop it into my cluster and the address resolution protocol (ARP) worked as advertised (pun intended) made me question if I'd missed something. I swapped my Contour service from NodePort to LoadBalancer and tested the assigned IPv4 address. Since it worked, I promptly updated the DNS records for the cluster hosted services and giggled a bit. That was easy. One stumbling block that I'm still dealing with is trying to get IPv6 working as well. While I easily got IPv4 routing working, IPv6 still evades me as I try to adopt IPv6 everywhere I can.
Next point of order was tackling kube-apiserver access. Because MetalLB is a Kubernetes service itself, I didn't really want to make accessing the API server rely on it (there's a GitHub issue about this). Much to my networking noob delight, my choice of Kubernetes Linux distribution supports VIPs out of the box! All I had to do was configure the VIP for the control plane nodes and it ✨ just worked ✨.
The final piece was Tailscale. I rely on Tailscale for accessing services while out of the house, and being able to route to Tailscale nodes directly is preferable to using a subnet router. Tailscale has a native Kubernetes operator, and while the last time I tried it, I did not deem it "production ready". Since then, it's come a long way, and setting it up for authenticating with kube-apiserver and as a LoadBalancer provider for the Contour deployment was trivial. The operator runs a pod with Tailscale within, which can be rescheduled between nodes as required - it's not quite HA in the sense there's multiple instances running at once, but I'm at least satisfied with failover.
And with MetalLB, Talos VIP and the Tailscale operator in place, I turned off the HAProxy virtual machine to see what broke. Nothing did.
I contemplated things for a moment after that, and it's what motivated me to really start digging into subjects like Border Gateway Protocol (BGP). Networking has always been a bit of a scary area for me, and the required complexity of IPv6 didn't help my mindset. I'm gradually overcoming this, and my homelab has been critical to my experience.
It's also worth highlighting the importance of DNS in this situation. I only had to update a few A and AAAA records, which other records CNAME'd to, so downtime was minimal and I didn't have to update any configuration beyond that.
And now the homelab cluster is humming along nicely, with more confidence that an entire server going offline won't impact access to the cluster, or cluster operation itself.