I tried building a home lab using Illumos distros - here's how that went

Ayush Pande is a PC hardware and gaming writer. When he's not working on a new article, you can find him with his head stuck inside a PC or tinkering with a server operating system. Besides computing, his interests include spending hours in long RPGs, yelling at his friends in co-op games, and practicing guitar.

Be it the uber-popular Proxmox, the enterprise-grade XCP-ng, or the NAS-centric TrueNAS Scale, there are plenty of options to choose from when you’re building a home server. But once you dig deeper into the home lab rabbit hole, you may end up encountering certain Unix-based operating systems designed specifically for virtualization workloads.

When I was searching for some cool alternatives to Proxmox, SmartOS and OmniOS stood out thanks to their Illumos roots. For those unaware, Illumos is a Unix OS that's based on OpenSolaris, which, in turn, is derived from Solaris. If you're curious about how these non-mainstream operating systems perform, here’s a log of my experiences after trying out OmniOS and SmartOS for a couple of days.

OmniOS: An ultra-lightweight home lab OS

With a CLI-heavy design

Since I’d heard good things about OmniOS, I was quite excited to test it out. Once I downloaded the ISO file, the first thing that stood out to me was the size of the image. Rather than occupying a couple of GBs like most modern operating systems, the OmniOS ISO was barely under 300MB!

After writing the ISO onto a flash drive, I was surprised that the installation wizard of OmniOS booted up within mere seconds on my old PC comprising a Ryzen 5 1600 and 8GBs of RAM. Unfortunately, this is where I encountered my first roadblock. For some odd reason, the wizard couldn’t create a new ZFS pool during the installation, and after trying out numerous fixes ranging from switching my flash drive and SSD to flashing an old version of the OmniOS, I finally got it working on my dual-Xeon server PC.

Unix commands, as far as the eye can see

While the installation proceeded smoothly, the real ordeal was about to start. That’s because I had to step out of comfy web UIs and dive headfirst into an OS that requires an entirely new set of Unix commands. Thankfully, the documentation on the OmniOS website came in handy when setting up SSH on the server, though I had to execute multiple commands to accomplish that. Next, I wanted to deploy a Zone. At the risk of oversimplifying the term, you can think of a Zone as the Solaris variant of a container, with a few discrepancies in the resource management aspect that set them apart from LXC, FreeBSD Jails, and other container types.

Unfortunately, creating a zone involved a lot of elbow grease, as I had to run multiple commands to initialize it. I executed another series of commands to deploy a VM using the Bhyve hypervisor, and after a few minutes of troubleshooting, I managed to get it running without any performance issues. Although Napp-it technically has a web UI available for OmniOS, I was unable to set it up on my system, meaning I had to learn an entirely different set of commands to use this platform.

SmartOS: An Illumos-based platform that lets you set up a convenient web UI

Though it lacks certain quality-of-life features

With the OmniOS tests done and dusted, it was time to turn my attention to SmartOS. Thankfully, the installation process with this Solaris/Illumos/Unix-based OS was relatively straightforward, and I had no issues setting it up on my Ryzen 5 1600 system.

I was also able to log into the system via SSH without setting up a TTYA port. Unlike its full-on CLI-based sibling, SmartOS lets you install a web UI with a simple command. And let me tell you, SmartOS probably has the cleanest-looking interface I’ve seen in a home lab/NAS OS.

Not to mention, the web UI provides easy access to LX, ZVOL, and Zone Dataset images, further simplifying the process of setting up the virtualized guests. That said, it lacks a couple of QOL features. For starters, you can’t backup the VMs, containers, or zones from the web UI. Likewise, there isn’t a shell window for the host machine on the interface, meaning you’ll have to switch to PuTTY every time you want to execute some commands. Unlike XCP-ng and Proxmox, it also lacks a built-in VNC console in the UI, so you’ll need to set up those tools on another machine to access your containers and VMs

Should you use Illumos platforms for your home lab?

Well, it depends. If you’re a complete beginner, I’d advise staying away from both and sticking to the tried-and-tested Proxmox. But in case Proxmox doesn’t meet your fancy or you want a clean-looking interface, it might be a good idea to switch to SmartOS and its neat web UI. Meanwhile, advanced tinkerers and DIY enthusiasts who love a good challenge might find OmniOS a fun alternative to Proxmox and XCP-ng. Either way, just be prepared to go through piles of documentation, as you’ll need to learn several new commands to make full use of these Solaris distros.