Kubernetes on Oxide: How customer needs shaped our integrations
oxide.computerI'm interested to see how the `oxide-cloud-controller-manager` is being built for "modern" Kubernetes and if it leads to any signify difference compared to CCMs that originated in-tree. Given the way Oxide engineer their solutions this could be really interesting.
FYI I've got `karpenter-provider-oxide` on my bingo card...
There are few paths we can take here. The CCM's primary responsibility is to implement the cloudprovider.Interface[0] which has node, route, and service controllers. However, the CCM can run arbitrary named controllers as well, giving us a fun extension point for the future. The in-tree CCMs are being phased out in favor of out-of-tree CCMs. The AWS Cloud Provider[1] is a good example of what that's starting to look like.
My colleague demo'd Karpeneter internally. We haven't committed releasing it yet but we're discussing it.
[0] https://github.com/kubernetes/cloud-provider/blob/master/clo...
My point was that the major cloud CCMs are rooted (at least spiritually) in in-tree implementations. With Oxide having an API first strategy (similar in principle to a major cloud provider) the direction you take on your greenfield CCM could be interesting.
RE Karpenter, it always seemed like a natural fit for Oxide, even more so than some of the currently implementors. And now you have the expertise in the team, I'd be really interested in the reason if you don't go down that path.
Karpenter is definitely a natural fit for Oxide. There's some interesting boundaries that we're discussing internally, like @sudomateo mentioned. One of which is CAPI. There's a CAPI Karpenter provider that elmiko built (notably back in the very early days of Karpenter). I think there's room for both use-cases. Some may want to use CAPI for everything and others may want a platform that is based on CAPI but guest clusters are not.
One implementation specific detail which makes Karpenter interesting on Oxide is the tunable CPU and memory parameters rather than strict instance type shapes. The prototype I built for Karpenter on Oxide generates all possible "instance type" combinations, so you can create some really specific nodes to bin-pack pods.
Another interesting area is multiple providers. This is becoming pretty common across public clouds too. A multi-provider Karpenter is something I'm interested in and I know some folks have already been gluing together, but the Karpenter story isn't great on maintaining those since you basically need to compile them together today. CAPIs multi-provider story is a bit cleaner since it only relies on CRDs.
If you have ideas, let's chat in the Kubernetes slack #karpenter-dev.
I'd assumed @sudomateo was talking about you when he said his colleague had built a prototype.
Totally, the out-of-tree CCM gives us a good opportunity to differentiate.
Autoscaling is a requested feature, and Karpenter fits that shape naturally. The nuance is to decide where something like the Cluster API provider ends and Karpenter begins since there's a bit of overlap in concerns. Specifically, both want to manage Kubernetes nodes but for different reasons. We're discussing it though. We have a Kubernetes watercooler meeting today where we'll likely discuss the comments in this post!
Yes this would be great addition! Also it would be awesome to see proper load balancer, ingress controller with `gateway api` and a `api gateway` (two separate things).
I’m not an oxide customer, just a fan. I have lots of ideas around how something like this should look, being disappointed with the complexity and also shortcomings of products on the market for this stuff, both in the cloud and on prem.
I'll need to record a video on what's possible today. My personal Kubernetes cluster running on Oxide uses the CCM for LoadBalancer services and NGINX Gateway Fabric for Gateway API things. I'm mostly using HTTPRoute resources today.
There's an opportunity to more tightly integrate at the network layer but we'll want to get our load balancer released first.
> My personal Kubernetes cluster running on Oxide
What a flex.
It's gotten to the point I'm starting to actively look for Oxide customers so I can apply with them.
I have seldom wanted anything as much as I want an Oxide rack at home. Maybe in 40 years we'll start to see the first ones show up in surplus auctions...
You can run the control plane at home. I have a video on how to do it. Until we make a smaller footprint that's the only way you'd get Oxide at home.
Oh, that's awesome, thanks for making it! I'll take a look.
Every single Oxide article has comments about people wanting one at home.
A great example of completely misdirected marketing and/or engineering. Oxide should have developed home microcomputers, and they would have sold like cupcakes, with their ASCII art marketing. Instead they do million dollar "mainframes" that no one (except VCs) wants to buy.
(I think you meant "like hotcakes", not "like cupcakes"?)
In any event: from the beginning, we have always been targeted at the enterprise buyer who is looking at annual public cloud bills in the tens, hundreds or (in some cases) thousands of millions of dollars per year. We love the enthusiasm that Oxide engenders among the home lab set, but that's not how we have geared the business (for lots of good reasons).
Also, for whatever it's worth: people do want to buy them, it turns out.
I’d really love to know a ballpark price on a minimal install without wasting anyone’s time, have you guys ever discussed real numbers on the podcast? I listen occasionally but definitely haven’t heard every episode.
This comment is a great example of completely misunderstanding… everything?
Just because a few people on a niche tech site are excited by niche tech stuff doesn’t mean you should focus your entire business on them, and the implication that you’d be successful in marketing a consumer product with ASCII art is mind blowing.
Homelab tinkerers are essentially a non-market. They barely want to pay for the cheapest commodity hardware. Tens of individuals would buy Oxide racks or theoretical microcomputers with their own money.
> Homelab tinkerers are essentially a non-market.
Sadly, I agree.
> They barely want to pay [...]
Not so! I've conservatively dropped $10k on my homelab so far, and have cobbled together a passable VM host/network/home-auto system in a half-rack. I would happily have paid more for an integrated (hypothetical) MicroOx! As fun as it is to play with cage nuts and to crimp cables, I love doinking around with a functional system even more.
There must be literally dozens of us out there...
Unfortunately yes. I would realistically pay about $10K for a MicroOxide rack, because that’s what I can justify spending on what is essentially a hobby. I know that’s never, ever going to happen.
i.e. enthusiasts. The cheapest, most demanding, the loudest and least loyal customers.
I'm sure they have done plenty of market research, and that research probably indicates that enthusiasts who buy server racks are a much smaller market than large enterprise users that need easily scalable on-prem resources.
Lots of people saying "I would love this" is not a profitable market. There are a lot of implicit assumptions in that phrase. Would you buy the lowest level rack for $100k, paying extra for any support? (I don't have any insight into actual pricing, but I know enterprise consumers don't bat an eyelid at prices like that.)
If not then maybe the "I would love this" is not relevant for a real market.
While I'm fascinated by Oxide, their style and work culture (if open documents are to be believed), I do think you're misunderstanding the homelab people a lot. Yes, I want the chance to play with an actual Oxide hardware in the same sense I'd want to touch a mainframe from the 80s. In the end though this is an extremely opinionated piece of tech that doesn't really invite or support tinkering. And homelab is about building something that no sane production-grade maker would ever be doing, so selling to this market would be quite an insane business venture.
Lots of people want to buy our stuff!
Favorited. Assuming this comment must be satire.
This is the perfect example of a delusional HN comment. None of it makes any sense, most notably the idea that VCs are buying any of this server hardware.
It's so bad I can only guess that it's purposefully so. "sell like cupcakes"????
fwiw, I am very satisfied with talos and k8s at home.
I run this on a TuringPi board at home. I wouldn't get the TuringPi again but Talos for home use has been great. The only issue for me is I have access to Oxide so I moved most of my home lab there.
I would absolutely kill for them to open source their documentation system.
Are you referring to our RFD system?[0][1] Because if so, good news![2]
[0] https://rfd.shared.oxide.computer/rfd/0001
[1] https://oxide-and-friends.transistor.fm/episodes/rfds-the-ba...
OMG, you just made my day!
does this mean that you won't be killing anyone?
They might still be available for killing someone out of gratitude
Sacrifices for future bountiful harvests
Why would a “doc system” use React?
Because it’s a web site.
Because HTML/CSS apparently isn't enough for documentation. /s
React is one way of generating HTML.
Why shouldn't a doc system use React? One word answers consisting of the word bloat are not going to be accepted, please use your words and be specific. The router sucks, the language is ass, give details!
It’s best for SPAs. A pile of docs and comments does not require nor gain much from it. i.e. a poor tradeoff, at first glance anyway.
Great post.
ClusterAPI never got the love that it should. I spent a good amount of time on it at VMware, as Tanzu heavily leverages it for cluster deployment (mostly CAP-A and CAP-V). It's basically kubeadm + the spirit of Terraform, Kubernetes controller edition. There's lots of great enterprise-ready options for centralized k8s cluster fleet management these days, but it works really well for folks that are all-in on GitOps and such.
Disclaimer: I'm totally biased here.
I talked with a colleague from Oxide in 2024 about your Kubernetes story and he said back then "not yet but soon-ish". Seems like soon-ish is now :)
We said we'd talk again when that happens but he's since left Oxide. If you (or well...your customers) are interested in a Kubernetes native data platform 100% open source we'd be very happy to talk about how that could work easily. As it's "just" Kubernetes it should be trivial but we'd be happy to test and add you to our list: https://docs.stackable.tech/home/stable/kubernetes/#supporte...
The offer stands. If you're interested, my mail is in my HN profile. https://stackable.tech
Love to see the CAPOx provider and buy in to Cluster API
Yeah me too, unfortunately I cant justify huge 100K USD servers so ill continue with cluster api provider hetzner and metal3. The whole CAPI ecosystem is quite cool.
Kubernetes is where Oxide, from my armchair, doesn't yet quite match the public cloud.
On AWS EKS Fargate, each pod runs in its own dedicated VM. With Oxide each k8s node is a VM, so you still need something like Talos.
On networking it looks like it is getting closer. Where you can have external subnet give pod routed IPs without overlay. But the gap, as marked by the article, is also the load balancing.
It would be nice if Kubernetes were a native feature out-of-the-box. Also integrated within the existing user/access control.
We're discussing what "native Kubernetes" looks like on Oxide in the limit. There's a bunch to build, some of which is blocked on product gaps. We'll get there though!
I don't necessarily want to match the public cloud experience if there's an opportunity for Oxide to exceed the public cloud experience. Eliminating the overlay is a good example of this. We have customers using external subnets to eliminate the overlay but we haven't integrated that into our controllers yet.
Bug report I guess, but the pages top navigation seems completely inoperable to me on my iPad. Clicking with my finger does not work. Clicking with my Apple Pencil does not work. Any sort of hover state that might be there does not work (with the pencil)
Reported internally. Thank you!
So you’d be attaching a new volume to the running worker VM for each PVC? This seems a little odd to me. Could you attach a single large volume, and do path-based provisioning on that?
> So you’d be attaching a new volume to the running worker VM for each PVC?
That's what we prototyped before local disk was released and before we started disk hot-plug work.
> Could you attach a single large volume, and do path-based provisioning on that?
Possibly. We'd still want disk hot-plug first. Otherwise, customers would have to create their cluster in a certain shape before using PVCs.
Does seem like that could be an issue for RWX PVCs, unless those volumes can be mounted to multiple nodes. Depending on the workload, you might be better off using NFS.
There is no one-size-fits-all, but I would say NFS is nowadays not a preferred option for most modern distributed datastores. They are fine with local storage only, and achieve coordination with higher-level protocols. Running them on top of NFS kills their performance.
I am wondering when would you use kubernetes on oxide vs running kubernetes with kubevirt on baremetal?
At first look, it feels like oxide is equivalent to proxmox or some virtualization tool, may be it uses qemu stuff underneath.
Just curious.
The reason I am asking this is that, we have lot onprem scenarios in our business. We are tightly coupled with k8s, to solve this we started building an internal project that is kubernetes API compatible [1] but runs containerd or WASM or our platform natively.
Just curious how oxides work in this scenario
Good question. We use our own hypervisor[0] that's not KVM/QEMU. We don't have nested virtualization today so we don't follow the KubeVirt model, though we are discussing what CRDs like OxideInstance would look like for those that want to operate solely in Kubernetes manifests.
The core primitive on Oxide is the instance (virtual machine). We could support some container primitive, but that's a larger product direction discussion. Our host OS is Helios (Illumos) so there are details to iron out there regarding what abstractions we would build and expose to the users. Not impossible but not something we're currently pursuing either given that we have other immediate product asks.
If you're at a scale where compute density, power efficiency, security, and rack-level API management matters then that's where Oxide makes sense for you.
Could there theoretically be a swappable hypervisor for oxide some day? I'm imagining a oxide Linux distro of sorts running on the hosts, and using kvm for vms, but still oxide control plane bits downstream. Then you get nested virt and gpu support, and could maybe even virtualize a traditional oxide host. Like running multi-host vcenter clusters inside a single esxi, amazingly powerful lab scenarios, teardown is even fun
You would use Oxide if you hate Dell/HPE/Lenovo/Supermicro. Also most people won't run k8s on bare metal because they want to dynamically provision a bunch of clusters.
You treat the hardware as a software problem and provision it dynamically. Also keeps I/O high if you have data locality (HCI).
you can do that on k8s
Oxide is shipping the baremetal server as well, with integrated hypervisor - so running kubernetes with kubevirt on it makes less sense than integrating with the host hypervisor
Yeah me too. Like I kinda compare oxide to coreweave and coreweave is built on top of k8s afaik. I'm wondering what the underlying tech oxide leans on. If I was doing this I would be using talos and k8s on baremetal and building on top of that. Container first rather than VM first seems like a big advantage imo. Most workloads will be containers.
Coreweave is a cloud provider right? I dont think it is apple to apple comparison.
What's a cloud? K8s is a cloud imo, so is what oxide is providing afaict
Cloud is an API you can hit to get access to hardware. Oxide sells hardware.
Oxide has an API on top that gets you services. The whole pitch is AWS with purchased hardware, no? That's cloud
> The whole pitch is AWS with purchased hardware, no?
HBOM
SBOM
Unparalleled support w/ vertical integration
Tons of companies will see you a "cloud". The other parts seem pretty unique.
Yes but you buy the hardware and have to rack it, and then you have a cloud at home.
Extreme nit: because you purchase a rack at a time, you don't actually rack it. You wheel it in, plug in network and power, and hit "on".
Most security minded orgs and also AWS deploy containers in VMs for better security
oxide and coreweave?
Oxide is in-house custom everything. Switches. Racks. Power bus. BMC. Firmware. OS. Software and APIs. Virtualization. Etc.
When oxide provide your metal?
How are folks managing stateful workloads like databases on Oxide?
I cover this a bit in the post. Today they are running something like Longhorn backed by Oxide storage. When we release our CSI plugin they can use that instead.
why longhorn and not mayastor? longhorn doesn't even have nvmeof that is not experimental
Could be whichever. Customers have expressed a desired for Longhorn given they were already using it so we started our testing there. Ultimately our CSI plugin will be the official Oxide path.
Why Mayastor and not (Rook-)Ceph?
Slow, buggy and resource hungry. Faster spdk-based engine rewrite has been on the way for like 7 years now
Buggy hasn't been my experience and the overhead should get smaller as storage grows. In any case, you'd get extra features such as S3, which would otherwise require running another service
One cool feature of ceph is reed solomon erasure coding, myastor only does replication if storage efficiency is important
4+2 Erasure Coding has only half the storage overhead of 3x replication but the same fault tolerance
Also re spdk
https://muratkarslioglu.com/blog/io-uring-spdk-kernel-bypass...
RS and XOR EC are definitely useful features for an object store that's storing from like tens of petabytes to exabytes worth of data but for a remote block device with a typical size up to a couple dozen terabytes I'm not so sure. Thanks for the link I will check it out
Edit: skimmed it, classic AI "analysis" - totally ignores all the crap io_uring went through
I think it'd be cooler if you could just run kubernetes directly on their bare metal hardware without their hypervisor in between
Then why pay them? There are existing vendors in that space.
I’ve tried so hard to figure out what the hell oxide even is, but the website is utterly impenetrable. I guess it’s an “integrated platform” and does “AI” and it’s “purpose built for frontier workloads”, but… I have absolutely no idea what it is, still.
I guess it’s a bunch of computers. It says “AMD” in a bunch of places so, I guess they’re x86. Where are the GPU’s? If it’s for “frontier workloads”, wouldn’t those be sorta important? Is it its own OS? No idea. I see people adjacent to Oxide mention IllumOS sometimes[0], so does that mean it’s a solaris-like OS? Why would I want to run k8s (presumably with linux containers) on a not-linux OS? Or does it virtualize linux instances?
Do actual CTO’s go for this type of marketing, with zero details and nothing but pure fluff about “solutions”?
[0] This of course is from HN comments, I see no mention of any operating system whatsoever on the website, so if it is IllumOS, it’s not like you can easily find that info anywhere.
It's a rack-sale computer that uses hardware and software co-design to achieve things a stack of disparate 1U servers and network switches can't. You plug in power. You connect it to your network. You get a cloud provider like control plane API and web console to provision VMs, disks, and VPCs. That's it papa.
The host OS is an implementation detail since you, the customer, aren't running workloads directly on the host OS. The VMs running on Oxide run on our host OS, much like other cloud providers.
That rack-scale hardware and software co-design allows Oxide to provide higher CPU and memory density in the same footprint at lower power than competitors. That's super helpful for companies operating at scale where data center power matters. The co-design allows us tackle security problems by eliminating the BIOS, providing attestation from the firmware up to the guest VM, and actually updating the firmware that's on Oxide rather than letting it rot like most customers do today.
This is why when people ask which verticals Oxide targets we kinda say "all of them" because different customers benefit from different Oxide features, but all customers need the core VM, disk, VPC abstraction. Some customers come to us because they don't have an API to manage their on-premises compute and Oxide solves that. Others come to us because they need absolute confidence that there's no malicious firmware running in their compute stack. Others come to us because they are tired of paying exorbitant amounts of recurring money just to run on-premises compute.
Our job is to make Oxide an appealing on-premises computing platform for customers to run their public cloud provider workloads and on-premises workloads without it feeling like it's an entirely different platform than you're used to.
I wrote this back in 2022, but it's still relevant https://news.ycombinator.com/item?id=30678324
It's a private cloud. I guess you might not be a target customer.
I develop cloud infrastructure controller software for a living but sure. I’m certainly not a target customer, but as someone casually interested in this space it’s certainly very light on details.
Re-assessing my criticism, I think I’m mostly complaining that I can’t figure out what software it’s running. But I guess the target customers don’t care, since it’s an implementation detail. I’ve only ever heard “oxide” in the past in reference to the OS they were (are?) developing, which a web search says is called “hubris”, so when I see an article like OP I’m wondering “are they running k8s on top of hubris? Wow!” But I’m imagining that’s very much not the case.
In the comments in this very section an oxide employee mentions they’re using Illumos, but I don’t see that anywhere on their website either.
But you’re probably right in that the target customer doesn’t give a shit what OS it runs, so long as they can get instances deployed (although I would question why, if you’re going to run k8s anyway, you don’t just run it on bare metal and skip the hypervisor, but that’s my bias showing up as someone who writes bare metal controller software.)
You're completely skipping the management of the bare-metal hardware itself. Sure, you can buy a stack of whatever commodity hardware you want and run Kubernetes on bare-metal. What about updating the firmware on that hardware? What about expanding and the hardware you purchases is longer available? Now you have more disparate hardware that may not even be compatible with the software you're running. Those problems exist at scale and is what Oxide is tackling. It's more than just buying a computer and set it and forget it.
I'm not really criticizing oxide as a product here, I'm sure it's great. I'm criticizing the website for being heavy on "solutions" and light on details. But really, the main detail it's lacking is the operating system it uses, so as I said, I needed to refocus the criticism.
But to your point, I don't think I ever said machines don't need a management control plane (I develop a bare metal management control plane for $DAYJOB, I'm fully aware of what it entails!) I'm saying that you don't need to insert a VM layer between the bare metal and containers: Having k8s run on the bare metal OS would be ideal IMO.
But since Oxide seems to be Illumos-based, that's basically not possible... At least not for containers as most people know them (ie. with linux-based docker images.) Heck, even if running a VM layer between the OS and the containers, it looks like running your own hypervisor prevents you from (currently) having nested virtualization, which makes certain k8s workloads a lot harder.
"Light on details" is not often a criticism that we get, but I think you might be looking in the wrong place: for the kind of detail that you're looking for, you should be heading into our RFDs -- specifically (for the issue you seem to be most interested in), RFD 26.[0][1]
(To answer one of your specific questions, "hubris" is an embedded RTOS and is what runs the BMC-like service processor, and illumos is the host operating system.)
Here's the docs: