Installing Ubuntu over 17000 KM distance using MAAS, VxLAN & Tailscale

During my work on MAAS (Metal-as-a-Service) I’ve spent a lot of time figuring out how to give people an easy and awesome experience as fast as possible.

When people hear about bare metal provisioning, if they’re in tech and infrastructure, I think they think “wow, that sounds interesting”. But I lose them pretty quickly because:

1. Not everyone has a lab lying around
2. Not everyone has 2–3 hours to spend configuring their lab

I’ve already written this tutorial which lets you try MAAS out relatively quickly, it still requires you to do it on an Ubuntu (or Windows 10 with Hyper-V) host machine.

This made me start thinking a lot, and I wondered about hosting MAAS in the cloud. At least that would mean MAAS itself doesn’t need any local hardware to run on. I’ve also recently written a Raspberry Pi tutorial for MAAS with some community friends, and started thinking about how that might fit in, too.

Small interlude! If you’re here, you might also like to know that I have started a homelab and self-hosting newsletter called “I Am the Cloud”, which you can find here:

If you like my style and this article, please go subscribe!

Warning — this actually didn’t turn out to be fast to do, but it did turn out to be fun to do! And, probably useful in case you want a MAAS in the Cloud somewhere, but want to break it out in different locations.

What we’ll end up with at the end is something like this:

What’s the 17000 KM thing about?

Well, once I got this working — I used it to install Ubuntu from a MAAS running in Australia, to a server under my desk in Stockholm, Sweden. Why? I don’t really know! Just thought it would be fun :).

In the screenshots and guides below, you’ll see me using Frankfurt and Stockholm, that’s just because it was faster to reproduce and write this article by doing that, but the first test was indeed from Melbourne to Stockholm.

Use the cloud, Jim

First up, I thought it would be nice to use a VM in the cloud. GCP, AWS, or anywhere really. Logically, I need to make this VM appear as though it’s locally present for any server trying to do a netboot. Most of my career has been spent in service provider networking, doing a lot with IP/MPLS, tunneling, mobile backhaul, and broadband aggregation. So I thought — I should be able to setup a virtual layer 2 network between a cloud MAAS instance and a local switch network.

With IP/MPLS making virtual layer 2 networks is like bread and butter, but in this case I don’t control the external networking for a VM. I decided to build a Layer 3 VPN using Wireguard, and run VxLAN over the top.

Once Wireguard is up, all that’s needed is to create Linux bridges at each endpoint, configure VxLAN, and connect the VTEPs to the bridges. At that point, any KVM could be joined to the bridge and should be able to netboot — it should be like MAAS and the VM are on the same layer 2 network. Alternatively, the bridge can be connected to a real physical interface connected to a switch, meaning that any physical machine connected to the switch could also netboot from MAAS.

Enough already, just tell me how to do it!

Alright already! This actually didn’t take too long to setup, thanks to my first foray into Tailscale. Whilst I’ve setup many a wireguard network (by hand), I wondered if by now there was better tooling. Well, Tailscale is that better tooling. VERY GOOD TOOLING indeed.

Anyway, here is what I setup. Pay attention to where the VTEPs and bridges are. I used an RPI4 to basically build a bridge box — it terminates VxLAN and bridges it to its ethernet interface. On the other end of the ethernet interface was a physical server, but it could have been a VM with a virtual interface.

Create a Tailscale account and set it up on your local PC

We’re going to want access to our Cloud MAAS, and the absolute easiest way to do that is just to run Tailscale on your local machine.

Go here, create an account, and log in.

Download and install the client on your machine, and follow the instructions:

When you’re finished, you should see something similar to the following, with your local machine connected. You can see the L3 VPN IP assigned to your computer.

Setup Tailscale on the RPI

Now that you have a Tailscale account, we can add your RPI or bridge to Tailscale too. This step assumes you have:

• An RPI with Raspbian or Ubuntu connected to the internet via its Wifi interface
• Access to your RPI via SSH

Log in to your RPI and install & configure Tailscale. The following commands should do it:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

Click the link generated when you bring Tailscale up, and this should result in your RPI being associated with your Tailscale account. If successful, you should now see something similar in your Tailscale console (note: my RPI vast-fish is “down” because as I write this, it was turned off):

Note the IP — let’s call it $RPI_IP. You’ll need it later. Here, it’s value is 100.68.73.100.

Create your VM and setup MAAS

This step assumes you are familiar with creating a VM in a public cloud provider. You can use any provider, but these steps show you how to do it with GCP. To make life easy for you, I’ve created a cloud-init.yml file that you can feed to the cloud provider at VM creation time.

The cloud-init file will:

1. Install and configure MAAS
2. Install Tailscale
3. Create a script for you to run after the VM is up that will configure VxLAN

When creating your VM, make sure you take a machine with 4 GB of RAM and 15–20GB of disk space. It’s up to you if you want to use SSDs, spinning drives, etc — but you do need to select Ubuntu 20.04 as the image for this guide:

The next important thing is that you provide the cloud-init.yml file. On GCP, you find the location to provide that near the bottom of the VM creation screen by expanding this tab:

Then, selecting management, and under meta-data adding a key value pair with the cloud-init script:

In this section you can also select a preemptible VM if you want to save some $ (just be aware it might get destroyed when you least expect it).
When you’re done, scroll down and click create.

Finish Cloud MAAS instance setup

SSH into your new VM, the easiest way to do this if you’re new to GCP is just to do this from the console:

Once you log in, you’ll find a script in /home/ubuntu:

Remember that $RPI_IP variable? We need that now when we run the script. In my case the value was 100.68.73.100. Run the script, providing the IP as an argument. When you get the authentication prompt for Tailscale, open it in your browser — this will add the Cloud VM instance to your Tailscale mesh:

When you’re done, you should see something like this:

Go to your Tailscale console and you should see your new cloud MAAS online (and at this point, I had powered on my RPI):

Right now, we have a Tailscale mesh up and running between our RPI and our Cloud MAAS. You can try to ping your RPI from your VM to verify that IP connectivity is working:

Notice the latency, looks about right as I am in Stockholm and my VM is in Frankfurt.

The script we ran on the VM also contained some VxLAN configuration and created a bridge br100. We haven’t yet created a VTEP or a bridge on the RPI, so there’s no way to test the VxLAN part is working yet. Add an IP to the bridge on the VM, ping it to check its all up and working locally:

anton@gcp-maas:/home/ubuntu$ sudo ip a add 192.168.3.1/24 dev br100
anton@gcp-maas:/home/ubuntu$ ping 192.168.3.1
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.050 ms

Time to jump over to the RPI and configure VxLAN and a bridge there. Sorry, no fancy scripts so you’ll have to resort to good old copy’n’paste. Make sure to update the local and remote IP addresses — local should equal the RPI’s own Tailscale address, and remote should be the Cloud MAAS VM Tailscale IP address. If you forgot what those are, you can find them in your Tailscale console or in your Tailscale client.

In my example below, I added a USB Ethernet adapter to my RPI, so I can expand my virtual layer 2 network to a switch or directly to a target server. In the example below, that interface is called enx2837370123a3. Yeah, I know it’s long and annoying :(. If you want a netbooting VM on your RPI using KVM, you’d need to replace this.

I also added a 192.168.3.2/24 address to the bridge so we can see if we can ping the VM via VxLAN:

sudo ip link del vxlan100
sudo ip link add vxlan100 type vxlan id 100  local 100.68.73.100 remote 100.64.139.101
sudo ip link set vxlan100 up
sudo brctl addbr br100
sudo brctl addif br100 vxlan100
sudo brctl stp br100 off
sudo ip link set br100 up
sudo brctl addif br100 enx2837370123a3
sudo ip link set enx2837370123a3 up
sudo ip addr add 192.168.3.2/24 dev br100

If you’ve done everything correct, you should be able to ping the the VM over VxLAN. The latency should be roughly the same:

ubuntu@vast-fish:~$ ping 192.168.3.1
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=42.0 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=41.0 ms
64 bytes from 192.168.3.1: icmp_seq=3 ttl=64 time=41.4 ms
64 bytes from 192.168.3.1: icmp_seq=4 ttl=64 time=41.1 ms
^C
 — — 192.168.3.1 ping statistics — -
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 41.011/41.369/42.004/0.389 ms

Congrats, you’ve now got a virtual Layer 2 network running from a VM somewhere (which could literally be anywhere), to a physical interface hanging off an RPI. Remember this? that’s what we’ve got now!

Since your local machine is part of wireguard, you can access the MAAS administration page directly from your browser. In my example, the address is the IP address of the Cloud MAAS VM: http://100.64.139.101:5240. The username and password is admin:admin. What’s that? It’s not encrypted? Well — it is, because you’re accessing it over wireguard, remember? And — nobody else can access that IP except you.

At this point, you can setup MAAS, plug in a server to the ethernet interface on your RPI, and provision it over the internet. Cool, huh? I won’t tell you how to use MAAS here, you can learn more about it at https://maas.io (in particular, the tutorials). About the only tip you need now is to ensure that you enable DHCP for the 192.168.3.0/24 network , by clicking “untagged” for that network under Subnets:

What else could I do with this?

Well, imagine you did something like this:

What else would I need to think about?

Well, I didn’t show it here, but there is a problem with internet access for the target servers. Once they’re deployed, they’d need a way to gain access. I haven’t tried it yet but there are numerous ways to solve it. I think an easy way is to use the RPI to provide a default gateway for the target machine. Perhaps there needs to be a separate VxLAN overlay/bridge for each site, and one subnet per site? Curious to hear what you think and come up with.

Wait — you never told us how well it worked from Australia …

Right! Well, it worked! It was quite interesting actually, because TFTP doesn’t really have any windowing functionality like TCP, so the initial boot part was quite slow. Even once the installation moved on to stages using TCP, it was less than ideal. I measured the throughput to be about 14 Mbps on average.

Would you like to know more about me? Check out traceroute.se, add me on LinkedIn, I’d love to connect!

I love bare metal Kubernetes too — if you want to see some other fun content I’ve made, check out my control plane slicing for Kubernetes video using MAAS (made for a CNCF webinar). The title is “Unleash the beast! Slash on-prem costs with Cluster API and MAAS for bare metal K8s” — probably worth watching :).