Our $100M Series B
oxide.computer715 points by spatulon a day ago
715 points by spatulon a day ago
I'm a Bryan Cantrill fan so I'm glad this is working out, I was extremely skeptical of them at the beginning(on HN too), I think because I've built DCs for many years and was stuck in a mindset that served my use case, I've come around to Oxide. My main concerns originally were 2 fold: "this seems bougie", is there actually a market for this, and, is there a good interoperability story with mix and match. From what I could tell the answers were "yes" and "don't care" - I had thought this wasn't a great answer but it seems I'm wrong. I was chatting with Boris Mann just last week about them and he said "actually John that isn't correct, think of how much quick compute needs to come online and how much discreet compute is going to be required with low management overhead, they're doing just fine and that market will grow" - After that I did some research and pondered on it for a day - I think my friend is right and I am wrong, I think at this point Oxide is going to be a really strong name and I wish them the best of luck.
I'm a fan just because they have such an incredibly good sounding product. like, it has no relevance to me, I'll never use it, but I get a deep sense of satisfaction just reading about how it works.
I was skeptical as well, if only because just being a better product isn't enough to win the market. Everything we hear about Oxide sounds like an impressive green field implementation of a data center, but is that enough? Do the people making buying decisions at this scale care if their sysadmins have better tools?
If it translates to improved efficiency, sure. And this big of a round seems to indicate that idea has some merit
I imagine most of the customers are highly technical, not typical generic business class.
> Do the people making buying decisions at this scale care if their sysadmins have better tools?
Look at who oxide is selling to and for what reasons.
It's about compute + software at rack scales. It does not matter if it is good it matters that it's integrated. Gear at this level is getting sold with a service contract and "good" means you dont have to field as many calls (keeping the margins up).
> Everything we hear about Oxide sounds like an impressive green field implementation of a data center, but is that enough?
Look at their CPU density and do the math on power. It's fairly low density. Look at the interconnects (100gb per system). Also fairly conservative. It's the perfect product to replace hardware that is aging out, as you wont have to re-plumb for more power/bandwidth, and you still get a massive upgrade.
As someone only tangentially familiar with this domain, I have questions about this:
> Look at their CPU density and do the math on power. It's fairly low density. Look at the interconnects (100gb per system). Also fairly conservative. It's the perfect product to replace hardware that is aging out, as you wont have to re-plumb for more power/bandwidth, and you still get a massive upgrade.
It sounds like the CPU density and network bandwidth are not great. If it's only suitable to replace aging systems, does that not limit their TAM? Or is that going to be their beachhead for grabbing further market share.
For a small company, a limited TAM isn't a problem (and honestly is probably an advantage) if the overall market is big. Datacenters as a whole are a ~$30B market per year. The last thing you want as a small company is a bunch of different customers pulling you in different directions. By limiting your TAM, you limit the number of problems you need to solve for a few years, and if everything goes well and you start outgrowing your TAM, you can expand later.
Is there a risk that the established players can commoditize oxide’s complement here? Is oxide’s product a feature that the big companies can just clone? I’m not sure to be honest. I have followed oxide through the news and am happy to see some progress in this area, I just want to know how to understand their success in the proper context.
I am not saying that I fully endorse the characterization of the parent, but it is true that we started selling these systems two years ago, and new hardware comes out with better stats all the time.
Given how small we are, new designs and refreshes take a while. Part of growing as a company is being able to do this more often. We'll get there :)
Datacenters seem to be increasingly power/cooling (i.e., power)/water (if they use it) limited. I'm wondering if the lower CPU density really matters when 75% of a DC risks remaining empty because the power budget is maxxed out already.
And yes the 1-for-1 replacement of older racks is probably a key selling point too.
What percentage of enterprise IT compute has not moved to a public cloud?
Only 30% have moved to the public cloud
https://www.goldmansachs.com/insights/articles/cloud-revenue...
I know I have been involved in multiple efforts to move the same workloads into and then out of the cloud, as corporate budgeting requirements prioritized either capex or opex at different times.
I'm not sure anyone really knows
uptime institute publishes some good numbers from survey, which puts on prem + colo still at >50% last I checked.
And still some additional 5% in like... on prem in closets.
Last year Amazon said it was 85% on prem. I dunno who has the right numbers.
Every single company I've worked with over the past 5 years has been repatriating from the cloud to their own DCs or colo.
The cloud doesn't pan out for long running, predictable workloads. Most companies are and will continue to use VMs for many years.
I must admit that I am much more unsophisticated than this, and yet I "invested" in Oxide (by running my own projects off Oxide servers), and it is gratifying to see them continue to grow. My (naive) assessment: (a) agreed with Cantrill's opinions on software, (b) liked his willingness to put himself out there, and (c) felt the eng blogs showed a high level of (socio-)technical ability.
I think for the internet to break out of walled gardens, high-quality independent datacenters need to exist -- nobody wants to manage their own datacenters, and nobody wants to rely on Google/Amazon/Microsoft's platforms or (even worse) business products. I hope this continues.
Did you order a unit from them or how did you manage to get access to the Oxide hardware?
I still dont get it. If someone else's software is running the hardware, what difference does it make if its on-prem or offsite?
> If someone else's software is running the hardware
Our stack is open source.
> what difference does it make if its on-prem or offsite?
The difference is not where it runs, it's that you own our racks, rather than rent them. In the traditional cloud, you're renting. Other vendors who sell you hardware will still have you paying software licensing fees, so it never feels like you truly own it. We don't have any licensing fees.
I like you. Are you guys hiring?
We are! https://oxide.computer/careers
I just have to say this is an incredible page. Everything is well thought out and there's no BS. The salary is upfront too and everything is remote. A gold standard for hiring pages?
Paying everyone at the company a flat rate of $207k, no matter their role or location, is quite the concept.
Are there any storage-related roles? Will Oxide redefine storage as well?
We don’t have any storage specific positions to my knowledge, but that falls under the control plane job, they’re the ones working on Crucible.
First let me say I really like this part of your guys narrative: you have really strong opinions about how infrastructure and IT should work at many levels, like technically and aesthetically, that seems real and nice and likable.
Focusing on just this financial narrative you're weaving, what stops a bank from selling "virtual racks" that work financially the same as owning an Oxide rack, but it's just AWS?
$1m buys you 42U of, whatever. You're handed an AWS account you do not pay for, but it has the $1m worth of, whatever in it, in perpetuity. Maybe the bank even throws in some fakey market you can "part out" and "sell" your rack to, years later, at some "market price."
It seems like, the product - and maybe the experience of buying the product - is what is most important to Oxide. It's really interesting to me, because I cannot wrap my head around what this narrative is:
You guys are Apple of Racks. But minus the iPhone, because there is no monopoly here. So, Apple (Minus iPhone) of Racks. Is that it? It's the rest of their offerings, which without the iPhone monopoly effects, are Buying Experiences. It's like when people buy $10,000 Mac Studios to "run LLMs", which of course they are going to do like, zero to one times, because they are excited about the idea of the product. For the audience that needs to "run LLMs" they buy, whatever, or rent. But they don't buy Mac Studios. Just because people do something doesn't mean it makes sense.
Is the narrative, AWS Doesn't Make Sense? AWS makes a ton of sense, for basically everyone. Everybody uses it and pays up the wazoo for it. And there are good objective reasons AWS makes sense, at basically all levels. Who is fooled by, "AWS doesn't make sense?"
The problem with AWS isn't even that they are expensive. It's that Amazon is greedy. It could be cheaper, which is a different thing than being expensive. It matters because "AWS stays greedy longer than the average Y Combinator company stays private" is an interesting bet for an investor to take. They could decide to be less greedy at any time, and indeed, it did not take long after offerings of S3-like storage from others led them to simply reduce prices.
What that is telling me is, I could take $100m in funding, sell $1m "racks" of equivalent compute on the Rolls Royce of cloud infrastructure, making everything financially and legally and imaginarily the same as ownership, and then take a $300k loss, right? On each "rack", same as your loss? It's a money losing business, but here I am making the money losing very pure, very arby. Is this what you are saying customers want?
Clearly they want a physical rack. By all means, I can send them a big steel box that provides them that aesthetic experience. Cloudflare, Google, they do the physical version of this all the time: dumb, empty appliances that are totally redundant, because people ask for them. RudderStack, Weights & Biases, a bunch of companies come to mind doing the same thing in software, like so called Kubernetes Operators that literally just provision API keys but pretend to be running on your infrastructure. People ask for Kubernetes operators, they made them, but of course, they don't do anything. They are imaginarily Kubernetes operators.
The reason there are licensing fees and rentals and whatever is the enterprise sales pipeline, right? Enterprise sales is, give people want they ask for. People ask for a price that's below $X up front, so that's what IT vendors do, and then it turns out people are okay with some ongoing licensing fees, so there. That's what they do.
So what IS it?
> what stops a bank from selling "virtual racks" that work financially the same as owning an Oxide rack, but it's just AWS?
I'm struggling to understand what you're suggesting here, to be honest. First of all, banks don't sell cloud compute, so no bank is going to do that. Secondly, what does "work financially the same" mean? These are fundamentally different products, AWS is a service, Oxide is purchasing hardware that you then own.
> $1m buys you 42U of, whatever. You're handed an AWS account you do not pay for, but it has the $1m worth of, whatever in it, in perpetuity. Maybe the bank even throws in some fakey market you can "part out" and "sell" your rack to, years later, at some "market price."
What would be the advantage to anyone in this arrangement? Why not just have an AWS account in this case?
> "AWS stays greedy longer than the average Y Combinator company stays private"
Just to be clear, we are not a yc company. But beyond that:
> The problem with AWS isn't even that they are expensive. It's that Amazon is greedy. It could be cheaper, which is a different thing than being expensive.
It is true that if Amazon dropped prices, then the "rent vs buy" equation changes for some customers. But there always will be some people for whom it makes sense to own, and some people for whom it makes sense to buy.
> RudderStack, Weights & Biases
Neither of these companies seem to sell general cloud computing? They also don't sell hardware? These seem like completely different businesses.
> So what IS it?
We sell servers. Customers buy those servers, put them in a data center, and get a private cloud. That's the business. Other folks are doing similar sorts of things, but they all tend to be integrating parts from various vendors. We believe that our product is of a higher quality, because we built the whole thing, from the ground up. Hardware and software, working together. There are other things that matter as well, but that's the big picture.
> I'm struggling to understand what you're suggesting here, to be honest. First of all, banks don't sell cloud compute, so no bank is going to do that. Secondly, what does "work financially the same" mean? These are fundamentally different products, AWS is a service, Oxide is purchasing hardware that you then own.
You're telling me it's important to people to "own" instead of "rent." Well I can manufacture an "Own" out of a "Rent": I write up a contract for my customer that says "$1,000,000 for 100 EPYC servers", I ship an empty steel box, the end user gets an AWS account which they cannot add anything to, and it has 100 EPYC server metal instances in it. I pay the bills in that account. Okay? Now I have created "owning" out of renting.
If we pontificate on the objective value of owning versus renting, such as the ability to sell the hardware, I can manufacture that too: you might want to sell your empty steel box 5 years later, and I will buy it from you for $300,000. Or maybe the value of owning versus renting is that owning is "cheaper" than AWS. Okay, I'll sell the rack for $500,000 instead of $1,000,000. Do you see?
The important part of course is, I didn't have to make any racks. I didn't have to write any software. I give people something they really, really want, AWS, and I give it to them in the shape of an "Own." Of course, your "Own" and my "Own" are different, but whose "Own" is more different from a typical IT purchaser? You guys know 100x better than me.
I agree that it sounds stupid though. That's BAD. If it sounds stupid to manufacture an "Own" out of a "Rent", and it is an arbitrage, and it also is something people like, that is BAD for you. If it sounds like something banks should not be involved in, that is BAD. Banks are involved in extremely lucrative businesses!
> First of all, banks don't sell cloud compute, so no bank is going to do that.
Ha ha, but this is what you do! You might not be a bank, but you are a $100m bank account. You're more bank than I am today. And you are selling something - you know, you say you sell a rack, and you say, "that's the business," and there are people on this thread - and this is not at all an unorthodox opinion - who are saying, what is really the material difference between cloud and on-premises compute, when the interfaces between the two look so similar?
A huge difference is "Rent" versus "Own" which is why we are talking about it and why you brought it up. But I am showing you that you can manufacture an "Own" out of a "Rent," and it would be interesting to see, well, should I take $100m and spend $200m on R&D with it (ha ha), or should I take $100m and directly fuel it into a flywheel of reselling AWS into a shape that people like, which is "Own" instead of "Rent"?
> Hardware and software, working together.
See it's stuff like this that says to me, "Apple (Minus the iPhones) of Racks." That is really your thinking. It's about buying experiences. You guys are answering this question, clearly, in your own rhetoric. Because if it was really about "Own" versus "Rent," a bank could do it.
Similar to what others noted earlier I'm having trouble understanding exactly what you're trying to communicate here. I'll respond based to points I am clear on.
Selling a customer a contract for on-premises computing and giving them a fake metal box and SaaS is borderline unethical depending on the terms of said contract. I understand the sentiment of that point though. There are many reasons a customer chooses to own instead of rent. Legal requirements, financial incentives, and even control over performance to name a few.
On-premises computing was so good that the cloud providers packaged it up and sold it back to people at a premium that could only ever be rented. The finances of that model don't make sense to many businesses as they look to reignite their on-premises computing with the modernity of the cloud providers. That's where Oxide shines in my opinion- being able to have on-premises computing that combines the efficiencies of the hyper scalers with an API-driven approach to managing resources. We take that a step further by building hardware and software in-house for additional benefits such as power efficiency, control over the networking stack, additional telemetry, etc.
there's seriously nothing complicated about this. Define ownership. Who cares about owning a bunch of computers? If you can get the compute, and you pay up front for it once, and then you can sell "it" later, and it's cheaper than renting, what does it matter if the compute comes in the form of a physical box or if it is in the cloud? So these are all things that matter about ownership! To me, occupying a physical space is not important for ownership, wrt servers.
Nobody is saying anything about anything unethical... I am mocking the idea of needing the steel box, forget about the steel box.
It's just Amazon Reserved Instances, but with an indefinite period. Okay? Isn't that an attractive product?
Why am I talking about banks? Because maybe in Oxide's deck it says, "Amazon will NEVER do this. Amazon will NEVER sell indefinite reserved instances." Fine. Well a bank can simply pay spot prices and sell you an up front price, if you want. Okay? It's the same thing. It only matters what Amazon does when we're talking about $100m Series B, which is what this article is about! It's not about the technology.
> On-premises computing was so good that the cloud providers packaged it up and sold it back to people at a premium that could only ever be rented.
No... guys... AWS makes sense. It's not a premium "that could only ever be rented." There are a ton of much cheaper cloud providers. Amazon just happens to be selling the Rolls Royce of clouds. They have a ridiculous margin. Figma makes more profit for AWS each year than it will ever make for itself in its entire lifetime. "that could only ever be rented" is simply not true, they can afford to make all sorts of innovative pricing models, reserved instances being one of them.
Oxide just hasn't had to compete with "99 Year AWS Reserved Instances." But absolutely, positively, utterly nothing stops them from offering that. They already give you a massive, MASSIVE discount for 3 year reservations.
That said, obviously not having to deal with human beings managing hardware is valuable. It's the same shit as the difference between "AI" meaning a computer and overseas workforces. They might produce the same outputs for the same cost, but think deeply about yourself: how much are you willing to pay to deal with a computer instead of an IT tech? To avoid phone calls? To avoid doing things that might be faster, but are in person?
You are casually transitioning between something that is pretty well understood through the history of online commercial computing (i.e. 1970s): 1) owned hardware versus leased/timeshared/rented hardware or services and 2) concepts of financial instruments that many readers here are probably not intimately familiar with.
If I needed a generator to run a remote mining operation, and you just told me to just buy energy futures instead, we'd be having a silly discussion. Whether it makes sense for me to rent or buy the generator has more to do with governments, [,tax ]laws, and risks that ultimately manifest as cashflow decisions. You have some valid thread you are pulling on for what are the economics of general purpose compute and to whom, but your argument needs a lot more care to carefully define and make your case and why it is okay to dismiss the outlier cases for instance.
> If I needed a generator to run a remote mining operation, and you just told me to just buy energy futures instead, we'd be having a silly discussion.
Exactly. Just because they are similar in some senses doesn't mean that they're fungible. Generator manufacturers still have a business even though you can purchase energy futures.
okay well this is an interesting and engaging conversation, if you guys someday make something in the $1,000 range you have a customer.
I'm not part of Oxide, but I think you're assuming everyone is okay with not controlling the hardware they run on.
There is plenty of addressable market where a public cloud or even a colocation of hardware doesn't make sense for whatever bespoke reason.
I don't think their target customer is startups, and that's okay. You likely aren't their target customer. But they've identified a customer profile, and want to provide a hardware + software experience that hasn't been available to that customer before.
It's not clear to me that the counter-party risk is _remotely_ similar between the two offerings. Someone reselling indefinite access to AWS could relatively easily set things up to go "poof" after a few years, whereas if it's your box, that's not possible. Now, critical components of the box could croak, and someone could still be screwed, but even there, you likely have more options for support than just paying Amazon's ransom.
yeah... AWS isn't going to set up anything to go poof, Oxide, despite making something awesome, will go poof if it charges too little money for it.
Amazon is expensive but not that expensive.
Oxide is selling a technical solution to technical customers with ownership needs or desires for security, regulatory, or other reasons. Access to the complete stack source code is another. I know a bajillion of these customers and they all have very, very deep pocket books.
They are specifically going after features in a way that no other vendor is, with an extreme care of execution of their crafts at the highest level.
The problems oxide is solving for these customers is something Amazon has shown no interest in. Could they? Sure, they could do anything they put their pocket books to doing, but they haven't.
> You're telling me it's important to people to "own" instead of "rent." Well I can manufacture an "Own" out of a "Rent": I write up a contract for my customer that says "$1,000,000 for 100 EPYC servers", I ship an empty steel box, the end user gets an AWS account which they cannot add anything to, and it has 100 EPYC server metal instances in it. I pay the bills in that account. Okay? Now I have created "owning" out of renting.
No, you haven’t. You didn’t deliver what was paid for. This wouldn’t be accepted.
> Do you see?
I’m sorry, but I do not. You’re not describing a business. You’re throwing some numbers out to describe a fraudulent enterprise.
> what is really the material difference between cloud and on-premises compute, when the interfaces between the two look so similar?
The material differences are around capex vs open spending, that you can locate your own hardware where you want to (which matters for things like “must be in a colo in southern Manhattan” (or any of the other reasons why physical location matters for latency reasons) or “must not leave the soil of $COUNTRY”), and the entirety of “TCO of owning is cheaper than renting for many workloads.” That’s just some of the larger obvious ones.
I have to agree with much of this, if you need something that feels like "the cloud" but on-premise then you could have used OpenStack for the last 10 years.
The only reasons to use Oxide racks are that you get an all-in-one solution and they don't charge you a subscription fee, you only pay upfront for the hardware once. But if this company goes public one day shareholders will surely push for a subscription based licensing model.
I have yet to see the benefit of "custom software" for "custom hardware". To me it looks like a liability, if Oxide stops to exist tomorrow you'll be left with a hunk of metal which is a dead end. The software being open source doesn't change that, if you have enough manpower to support such software on your own then you can surely support any other more flexible solution.
I wish them all the best but yeah I can't see any reason why someone would pay oxide over aws onprem rack solution.
I'm sure they'll find _some_ customers but they're going to be few and far between.
Banks care _a lot_ where the data is store, hence most banks are inclined to keep customer data on-prem.
Because data must be on-prem, banks are stuck in legacy infra paradigms. The whole org suffers, innovation is stiffled, yada yada…
An on-prem cloud product (hardware+software) is a game changer for these companies, IMO.
My question to oxide: how easy is to integrate external hardware into the cloud? For example: bunch of GPUs or a bunch of next-gen hardware like SambaNova.
Is the software open-source with reproducible builds of any runtime binaries?
Oxide has been remarkably transparent about the development and architecture of critical system components. We can only hope they succeed and inspire others to follow their transparency lead.
Open source is a requirement but not the only one. There are countless examples of companies building integrated solutions based off of open source projects that, when they went bankrupt, there was nobody to pick up the available pieces and continue moving the stack forward. Just pointing out that open source is not this magical escape hatch that some people think (at least not in corporate environments).
Especially so for Oxide's decidedly non-Linux setup. They are in a niche software ecosystem with practically no one else. Apparently mostly because they're practically all ex-Solaris staff.
https://www.illumos.org/docs/about/who/
(Listing all projects using ZFS or DTrace as "who uses Illumos" is cheating.)
I remember many Linux fans saying that monocultures were bad until Linux became so popular that Linux was the one benefiting from a monoculture. Despite that, the rationale against monocultures still applies.
That said, Illumos is influential as an organ donor to many others. There are a number of awesome technologies in it.
Oh I would love to have some healthy competition to Linux, but I am not rooting for Solaris to do that, I'd rather have one of the Rust-based microkernel actually git gud. Time to shake the foundations of the age-old security and isolation models, not resuscitate a dusty old thing built on piles of C and shell on top of a large monolithic kernel and pretend everything's fine.
Well, good news: we have one of those too![0]
Oh I am well aware. But I am hoping to run dynamic workloads, including virtual Linux machines, on a PC. It's a bit of a different world.
Latest one still in my to-read pile: https://lwn.net/Articles/1022920/
You want to run dynamic workloads on a PC? As in a desktop PC? That is clearly a completely different market than Oxide serves.
Or do you mean PC as in rackmounted servers? If that's what you meant, PC is a very poor word for it. That's kind of the point Oxide made from the beginning. Why are you running server workloads on a PC with a funny shape? Why do you need 84 power supplies (2/shelf) in your rack? Why do you need any keyboard or graphics controllers? Why don't you design for purpose a rack-sized server?
Or did you mean exactly what you wrote: "a PC"? You only need one server, not a whole rack's worth? Again, that is not the market Oxide is targeting.
Or you need to be able to run "dynamic workloads" that could require 40-4000 CPUs? You need hypervisors and orchestration, etc.? And you don't want them to be Solaris, or to run on Solaris? And you know all about Hubris and you don't want that either? But you think it would be nice if they weren't Linux? Maybe if they were modern microkernels written in something like Rust? But not the Hubris microkernel written in Rust?
I'm going to have to take you at your word. Your needs are "a bit of a different world" than Oxide fits.
But it's pretty cool that you still got some friendly personal attention from two big-name Oxide employees who seem willing to try to help you if they can. If you ever do find yourself in a world that aligns with theirs it appears that they are willing to try to accommodate you.
We're talking about healthy competition for Linux, Rusty microkernels, and I'm saying Hubris is not what I'm looking for because of the stated reasons. Hubris workloads are defined at build time and it does not target x86.
When I say PC I mean the large ecosystem of compatible performant hardware that exist out there, as opposed to e.g. RISC-V at this stage.
> Apparently mostly because they're practically all ex-Solaris staff.
I absolutely do not have a Solaris/illumos background! The first time I ever sshed into an illumos machine was my first day on the job.
Cloud computing is significantly more expensive than self-hosting for most large organizations. People are slowly figuring that out, and Oxide was a bet on the timing of that realization.
That's almost always been true but servers are more like commodities at this point.
iphone is nice and upgrade from commodities phone and I have one, but I wouldn't care much if my fridge has sleek UI because I just need it to be a fridge.
Even if it makes no sense as a technical thing, businesses will buy it. Look at all the huge companies that keep spending millions trying to DIY their own datacenter for the 3rd time. Enterprise loves to buy big iron and self-hosting crap, so I'm sure they will be successful selling this. However, I think they're going to need to branch out to more services in order to continue increasing their revenue every year (after 5+ years lets say).
> Look at all the huge companies that keep spending millions trying to DIY their own datacenter for the 3rd time
You mean all the huge companies that ran multiple datacenters before the cloud was even a thing?
Everyone at Oxide makes the same salary:
>We decided to do something outlandishly simple: take the salary that Steve, Jess, and I were going to pay ourselves, and pay that to everyone. [https://oxide.computer/blog/compensation-as-a-reflection-of-...]
Does everyone at Oxide have the same equity grant?
> Does everyone at Oxide have the same equity grant?
I thought I saw this question answered in a previous thread and the answer was basically “no”, but the question has been avoided a lot.
Aspects of equity compensation would inherently need to be different over time due to valuation, fundraising stage, and so on. However I always thought it was strange that they made a big deal about paying everyone the same base salary but then were silent on the equity comp strategy. Everyone knows that in a job like this the total comp is important.
The old Oxide compensation discussions were interesting. There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically. I heard this from a now ex-employee of Oxide who was describing how to navigate their hiring process, so take with a grain of salt.
EDIT: I checked their website again. The compensation link goes to a blog post ( https://oxide.computer/blog/compensation-as-a-reflection-of-... ) which has this section about equity:
> Some will say that we should be talking about equity, not cash compensation. While it’s true that startup equity is important, it’s also true that startup equity doesn’t pay the orthodontist’s bill or get the basement repainted. We believe that every employee should have equity to give them a stake in the company’s future (and that an outsized return for investors should also be an outsized return for employees), but we also believe that the presence of equity can’t be used as an excuse for unsustainably low cash compensation. As for how equity is determined, it really deserves its own in-depth treatment, but in short, equity compensates for risk – and in a startup, risk reduces over time: the first employee takes much more risk than the hundredth.
Which doesn't answer the question.
> There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically.
I've heard this from multiple hiring managers and C levels. The cognitive dissonance is amazing.
Do you know why I show up and work? Because I am paid for it, and in this country, medical is also gatekept by employment.
If I wasn't paid, I wouldn't work for them.
But somehow, I'm supposed to not care about money at the same time caring about money.
Back when my salary was shitty I cared about the money a lot.
I still like the money and the number going up. But now that it’s above a certain multiple of what I consider a “comfortable” life, I’m not worrying as much.
Would for change your job for an extra 500k a year? Surely right? An extra 50k? An extra 5k? An extra 500 dollars a year?
Theres some mental calculus that includes everything. Some people will just take “offer with most money” every time. Hell of a lot of people don’t.
I do think that C suite folks might not have the right vision of what “normal/comfortable” is for employees though. So they might offer a low number thinking it’s actually a “comfortable to live with at this stage of life” number and then get confused why they can’t recruit good talent and think people are obsessed with money.
First offer has gotta be within the ballpark of the right number if you don’t want your interviewee to immediately come back to you with a much higher number!
> I still like the money and the number going up. But now that it’s above a certain multiple of what I consider a “comfortable” life, I’m not worrying as much.
I agree, but that number for most people is not the $207k/yr Oxide is paying.
For most people that number is likely north of $500k, if not single digit millions.
I used to think (like 10 years ago) that $100k a year was all the money in the world and I'd be able to own a nice house and drive a nice car in Cali.
Now my household income is double that, and I'm not a homeowner (I admit I do rent a nice house in a Norcal suburb) and because of remote work I'm not too concerned about the car I drive (but my wife does drive a Tesla), and after emergency fund savings, retirement, bills, and helping family out, my wife and I are still somehow still paycheck to paycheck.
I'm not sharing this to complain, merely to just express that yeah, $207k in California can be comfortable, but it's not to the "worry not about money."
$200k salary for a single person living in the Midwest? That person can probably retire early in a few years.
Yeah my point is that it’s comfortable enough.
If you are growing your emergency fund then you aren’t really living paycheck to paycheck right? You’re putting money off to the side. Tho perhaps you’re not putting aside much for, like, travel.
Plenty of smaller startups not hitting that level, especially when they’re lean. And I think up until this raise Oxide was lean!
(To be honest given this finding raise I could see a salary increase for oxide across the board, to help with that)
That's true, but I still consider 3-6mos emergency fund to be a basic necessity, since living in the startup world you never know when you may need to use it.
A while back I interviewed for a startup that was asking me to work from 6am to 8pm for about 75% of what I make right now working regular 9-5 at a different startup. It was super early stage so that level of commitment was understandable but I can't imagine very many engineers with families willing to take that much of a paycut. But that's the name of the game, right? When you're early stage you can't afford to pay much but you need your product done yesterday. Thats probably why most startups fail. Cheap quality talent is hard to come by, at least stateside. Offshore developers can be cheap and amazing (someone living in India making $100k USD would probably be considered royalty) but if you don't have someone from that part of the world to vet the devs it could end up being more expensive in the long run.
> For most people that number is likely north of $500k, if not single digit millions
I think if your kind of comfortable is needing to clear 500k or million in comp a year then you aren’t not in scrappy startup mode! This is fine but money out the door is money that then needs to get raised in early rounds
I dunno, I do think oxide for basically everyone who joins is asking for them to get pay cuts but I would really hope that people would still at least putting some stuff into savings.
And like … even if the equity is variable if people are getting even a bit of equity, that might end up as something in the end.
Saying this I think with the recent raise the comp could be made higher just to make the buffer even better. There’s definitely an opportunity cost
That heavily depends on where you live and your life situation (young kids? older kids? No kids?).
It’s a bit of a class thing, isn’t it?
Independently wealthy folks can stay in the game longer without needing to extract a lot of cash compensation in the company’s early years.
This is true in the abstract, but we're talking about a salary of $207,264 in this case. You don't need to be independently wealthy to make it on our salary.
It's still a class thing. With the CoL in most SWE-heavy metros and the outright war on engineer pricing power over the last few years?
Just about any engineer who isnt independently wealthy is:
- recovering from layoffs and a period of unemployment at a high-COL cost structure
- facing layoffs with the above salients
- facing an imperative to insulate their fucking house with cash for their turn in the above crosshairs
Competence and a willingness to work hard stopped equalling access to basic necessities in November of 2022 - February of 2023, depending on how you count, and inflation has savaged a notional 200k.
"200k is plenty to live on" is rich, well-connected guy talk in 2025.
> With the CoL in most SWE-heavy metros
They’re a remote company. They’re not targeting HCOL metros.
> and inflation has savaged a notional 200k.
> "200k is plenty to live on" is rich, well-connected guy talk in 2025.
Claiming that $200k is not enough to live on is rich-guy talk. The majority of the country lives on well below $200K.
I agree that a single person living in a HCOL metro like SF Bay Area would not find this compensation attractive, but that person also has nearly infinite other jobs to choose from nearby.
Jobs like this (remote work for interesting startup) do not need to pay HCOL comp.
Nothing about a remote gig in one job affects the imperstives I highlighted about past layoffs or future uncertainty: pricing relocation into or out of e.g. SFBA at zero is another conversational gambit popular with the out-of-touch. You don't teleport.
It's bad generally, this is a terrible time to be a working person in general. How SWEs stack up against profession X? Depends on profession X.
SWEs are in a job market where a bunch of folks who have been repeatedly prosecuted for wage fixing (most successfully/recently in 2012) are taking another swing at it in a much more lawless regime. That's as precarious as it gets when this cabal monopolizes the front row at the Inaugeration.
> pricing relocation into or out of e.g. SFBA at zero is another conversational gambit popular with the out-of-touch. You don't teleport.
I think this is missing the point.
The target audience for remote jobs that pay $200K does not overlap with the target audience of people interested in working in SFBA at FAANG-level compensation.
If someone is interested in relocating to SFBA, they should do that and get a job there.
> "200k is plenty to live on" is rich, well-connected guy talk in 2025.
The median US income in 2023 was $39,982 for an individual, and $78,538 for a household.
My issue isn't with Oxide's comp steucture which I neither understand nor care about.
My issue is with made-for-life, self-appointed elder statesmen on HN saying anything at all about other people's finances and imperatives from the comfort of a study in whatever isyllic community they're pontificating from.
Just, be grateful, and say nothing.
> My issue isn't with Oxide's comp steucture which I neither understand nor care about.
This is a sub-thread about it, so that's what we're talking about though.
> Just, be grateful, and say nothing.
That's not how message boards work. If you say that it's ridiculous to live on such a salary, I'm going to point out how out of touch that is.
That's not what I said, you know that's not what I said. I said people who talk like that on HN are doing so from an unrelated personal experience.
This sub-thread is about the habits of speech that make a certain breed of armchair public policy economist jump out like a lump on plate glass. Get Patek in here and we could start a convention.
> you know that's not what I said.
Okay, just so you know, I did not understand that you were speaking in generalities whatsoever.
Fair enough, this is a topic with a high and asymmetrical charge quotient and misunderstandings are a fact of life in such.
I also regret any degree to which I've singled you out for a generally regrettable trend among the long-time community members: it's very easy to see the world through the lens of one's own experience and I've been as guilty as anyone of doing just that on plenty of occasions.
We should all strive for empathy and understanding, that goes double for people like you and me (and Thomas) who have been around forever.
200k a year. In US dollars. American dollars.
Now you have families, debt, etc. There are things. Minimum expectations for family etc. But come on! It’s not poverty wages!
“I am willing to take a pay cut for a thing that I’m passionate about” is such a normal thing that everyone serious I know in this industry says. Or like… even just ethical choices to leave money on the table (there’s a reason online casinos pay their software engineers so much!). Not everyone makes the choice (and I get people saying no) and but in a sense I gotta imagine it’s part of the calculus for making it work. “We won’t have to pay people half a mil in total comp like meta has to, because the mission is more straightforward”. I feel like an O&F ep mentioning someone from intel expecting _triple_ the comp. 600k!
And like… people saying it’s not enough and talking about equity. You’re not paying rent with equity!
Signed: a guy who was at a small startup and who would have been very happy with the inflation/CoL equivalent of 200k instead of what I had those early leaner years
At one point “joining the scrappy startup” does involve some scrappiness. Otherwise you’re just working in a division of Google that hasn’t been integrated into the borg yet.
Focusing on the raw dollar amount is a red-herring that always comes up in these conversations. 200k is plenty! is a sleight of hand. If I am building something with 10s of millions of value and you give me 200k and expect me to shut up cause it's plenty, you are deceiving me, full stop.
The "we all get paid the same" is a dishonest by omission. They don't all get paid the same, and while the peanut gallery may think so, I sure as hell don't think the IRS thinks the same way.
I personally don't really care what they pay their engineers, but to pretend to have this egalitarian approach to compensation and then hide the equity numbers is dishonest.
In other threads, the voice of HN tells us to consider equity in a startup as worthless and only ever count on base salary.
Here we instead hear that equity is as important as the air we breathe.
Yes, 200k is not life changing but at least it’s not pretending to be, like many other inflated equity schemes that never vest. It’s just an honest decent salary, better than what you can find in many places of the world.
I completely agree.
Note that according to https://www.usinflationcalculator.com/ 200k in 2025 is equivalent to about 160k in 2020 when the pandemic began, and 160k would've been on the low end for an experienced software engineer at a Series A or B startup at that time, in the SF Bay Area.
Exactly. It’s also important to know that they support remote work, so these salaries really are not bad at all. For someone in a non-tech city who doesn’t want to move this would be a great startup job.
My only reservation is that I’ve interviewed with or worked for multiple companies that made some claim about paying everyone the same and there was always some loophole: The company might have a fixed base pay but then use very different equity grants. One company claimed to give everyone the same base comp and equity but then it was discovered that some people were getting huge annual “guaranteed bonuses” that were effectively base compensation. It has left me tired of seeing companies push the idea of everyone being paid the same while not being open about the entire compensation structure.
EDIT: To be 100% clear, I don’t know what Oxide’s entire comp structure looks like. The examples above were for past companies I worked for.
We have salary and equity, no bonuses. Salary is identical, equity is not. I certainly agree with you that that kind of shenanigans can be annoying; I had a former employer who hired people with "oh it's a bonus but you always get 100% of it" and then, we did not get 100% of it.
> Salary is identical, equity is not.
Thank you. This is the question I was hoping to see an answer for.
More important than just getting answer, do you understand _why_ every employee isn't granted the same amount equity?
Isn’t it obvious? This is where they vary compensation for the same reasons everyone else varies compensation.
If they gave everyone the same equity compensation presumably they’d put that front and center like they do for the base salary.
The claims about paying everyone the same are a red herring because it’s only about base salary.
Because if you want to poach talent then you'll likely need to induce them to leave their existing employer and will need to match their existing equity grants that might soon vest. Obviously that number is different for different people.
Bonuses are bullshit. When the first startup I worked for was acquired they asked everybody to basically agree they won't quit for a period of time - transition. Fair enough. However for the ordinary staff (like me) this agreement had no actual carrot, they're offering a so-called "Bonus scheme", but it's clearly designed so that they decide what it pays, whereas execs are getting an up-front specific financial inducement to stay. So I explain to colleagues that if you might want to quit you should not sign, the "bonus" is worthless but you're locked in by signing - however I did get some pushback from people who could not see this or haven't done this before.
Sure enough when that bonus was due to pay out, I got a heads up from a friend (who was being compensated because he was like CTO or something) that it was worthless, because of course they get to pick the numbers so they're going to pick zero. I don't care, because I was on a different scheme and anyway I work for salary, if you want to pay me to do something, you can pay me, don't fuck about with nonsense about a "bonus", but some people ended up stuck for a year or two believing they're getting a bonus to wait.
It's bullshit to talk about salary equality as though that mattered when equity is the real upside, and you know it. The claim to strive for a "generational company" is just "we're a family here" in other terms. Congrats on the raise.
I think there are tremendous benefits to having a flat salary structure, regardless of upside. My other friends in tech are envious of the lack of having to waste time doing perf, for example.
> Congrats on the raise.
Thank you.
From your perspective of course it has benefits. From mine? Getting stack ranked wasn't even in my top 10 as concerns went, but I'm really good at my job, and I keep quality notes that make it easy to recap a quarter or half. (But like I said, I'm good at my job.) I don't really see what that has to do with focusing your rhetoric on 10% of startup comp as though it were 100% - especially now that your tax discount on cash comp has been restored! - but you're welcome, of course.
> I keep quality notes that make it easy to recap a quarter or half. (But like I said, I'm good at my job.)
The point is not that my friends don't do a good job, the point is that this is work that does not actually further the organization's goals directly, but is necessary in order to keep their job. They'd rather be doing the actual work they are hired to do.
Oh, come on. Administrative overhead is a reality of business everywhere and at every level; the idea that to manage an organization somehow necessarily impairs it is absurd. So is purporting falsely to eliminate that overhead on behalf of others, when basic professional competence instead involves for oneself learning to minimize it - to dispose of it, not by panicking or catastrophizing or sweeping it under a rug, but instead in a fashion such as that I described ie efficiently. To claim otherwise is infantilizing nonsense. It's fundamentally dishonest, though I grant you probably have never before so directly been told as much.
I moved from Meta, infamous for its performance reviews, to Oxide. The culture difference is night and day. The level of self-interested behavior seen at Meta just doesn't exist here.
By the way, I received every rating from Greatly Exceeds (including an additional equity grant) down to Meets Most, and the rating I got overall had very little correlation with either effort or impact. I got Meets Most for some of the most valuable and industry-impactful work I've done in my career, and Greatly Exceeds for something that got replaced in a year.
>I moved from Meta, infamous for its performance reviews, to Oxide. The culture difference is night and day. The level of self-interested behavior seen at Meta just doesn't exist here.
The culture difference between Meta and any startup will be night and day. People who are self interested min-maxxers don't join startups. Not dealing with "corporate politics" has to be in the top 5 reasons anyone leaves FAANG to join a startup. That has nothing to do with Oxide's comp structure.
> People who are self interested min-maxxers don't join startups. Not dealing with "corporate politics" has to be in the top 5 reasons...
Oh, sure! Now tell me another one. The idea that startups don't have politics is - well, I'll say it is extremely comedic, and we'll leave it at that.
Think about it for a minute. I'm not questioning the existence of the pipeline here described, and no one is questioning the existence of many pressing reasons for anyone at the FAANG "top of funnel" to want to flow along that pipeline about as quickly as is achievable.
But those "reasons" have effects on the people who experience them, because humans have emotions and psychologies and other such inconvenient externalities, and for like cause those effects are not instantly and perfectly ameliorated in every case by a simple change of environment.
Can you not straightforwardly see how this might produce some extremely adverse results, in a social and sociological sense? And how overt, documented, attributable, and discoverable personnel processes, far from some unreasonable burden, might serve a broadly beneficial role in such circumstances?
This is a reasonable argument. But look, I have my views based on my experiences and things I've heard from colleagues and friends, and you have yours.
Well, sure. That's Meta, the model for much of the industry, where that isn't the likewise and just as deservedly infamous Amazon. So when you say Oxide is better, I'm sure I can believe you that it is, but can you see why that still might not convince? It's like if I say I'd rather be beaten than stabbed. Obviously this is the sensible choice to make among the selection given, but still the question might reasonably be asked: can there really be no third option?
The issue is, you're speaking from the position of the organization itself. Yes, staff work is just as important as line work. The issue with perf isn't that it's staff work, it's that people who are ostensibly hired to do line work are forced to do staff work just to keep their jobs. And sure, you can argue "tough, that's just life," but it's not hard to see why people resent it. They want to be writing code, not putting together promo packets.
Anyway this is mostly just one example, it's just one that comes up often when I speak with my peers about how Oxide works vs other companies.
I see why people resent it; I'm saying they're foolish to do so. Why should I not seek involvement in staff work that determines so much of my future? Why should I not make myself responsible for the conduct of the business within the scope of my role, rather than just the parts which I happen to like and enjoy?
And since you can't seriously mean me to believe performance is not evaluated at Oxide, I really can't see how I'm meant to take any of what you're saying at face value. Instead it seems something much akin to "don't worry your clever little head about the boring ol' money stuff, darlin'! Don't you trust me to take good care of you?"
> I see why people resent it; I'm saying they're foolish to do so.
Okay, sure. I am also not a "I only want to put my head down and code" person either.
> And since you can't seriously mean me to believe performance is not evaluated at Oxide,
Not formally, no. Because there are no levels, no corresponding salary bands, there's no need to have a formal process, with all of the justification work that has to go in from the employee, and all of the reading and evaluating all of that stuff from management.
It is true that if you don't do your job, you'll be let go. However, that's a conversation that would happen between you and Bryan/Steve, not an annual or quarterly process with all of the paperwork and such that those formal processes demand.
Instead, we simply do our jobs, and get paid our salary.
It sounds like it isn't an environment for you, and that's okay.
> Instead, we simply do our jobs, and get paid our salary.
And some of "us" have an ownership stake, and some of "us" do not. But "we" like to talk about how everyone's working on a level playing field, anyway.
You're quite right. It isn't an environment for me.
Small nit, everyone has an ownership stake, it's that some are larger than others.
The clarification is welcome, inasmuch at least as it's good to know there are no missing grants. It doesn't really surprise to hear some are larger than others, which is invariably the case, usually by one or two orders of magnitude. I assume you're satisfied with yours.
I wouldn't wish to be taken as saying no employee is ever more valuable to the business than another. That would also be absurd. What I don't understand is why all the circumlocution.
> but I'm really good at my job
How good you think you are at your job is pretty meaningless if some manager above you wants to weaponise the company's perf policy against you.
I'm good enough at my job to see that when it's happening. That's one reason why I didn't say 'I think.' Another is because I know the way to bet is that, with this weapon forbidden, others will be found to replace it. Why go into any of that without even the promise of hazard pay? I'd rather just do honest work.
This "job" that you speak of, that you are so good at. Are you... at it now?
> This "job" that you speak of, that you are so good at. Are you... at it now?
Hello, Bryan! No, I'm on my own dime, no one else's. But it's decent of you to show such concern over my situation, in these uncertain times.
>It’s a bit of a class thing, isn’t it?
reminded - the "amateur" (vs. paid "professional") requirement in Olympic and other sports, i.e. participation for the sake of only pure sport spirit, back then came from the aristocracy who could invest a lot of time in those sports without having to take care of making a living.
And when wealthy execs and founders tell what they want people who is interested in the mission/vision/whatever other than money - well, it is the same class thing. The ones who need money naturally can't have the required purity of vision as it is clouded by that lowly need for money.
As long as the equity grant is the same as the founders, I as an independently wealthy person am happy to join the team that gives everyone the same salary.
Because, you know, you better have the same amount of skin in the game as I do.
> If I wasn't paid, I wouldn't work for them.
That's great, but useful to know not everyone thinks the same. When I transitioned to software development (from basically random "whatever pays my rent" jobs), besides my first software job, they were all because I liked the particular product in some way or another, and what the compensation was is basically the least interesting thing for me.
Of course, some level of base payment is needed, because I still needed to pay rent, but if I was choosing between two jobs where one was utterly boring but paid 3 times more than a fascinating job, I'd take the latter in a heartbeat. And no, I'm just an individual contributor who wants to like what I work with, not an executive, manager or similar.
The hypothetical situation you set up is interesting, in that past a base amount of money to survive and thrive, that you would choose the more intellectually stimulating position. And I do get that.
For me, if the hours were equal, I would choose the higher paying one. And then, I would create and make outside of work. And since I have that much higher wage, it could be a jump start on my own business.
And, enough money can buy independenance in that you can get this flexibility of doing as you choose.
> For me, if the hours were equal, I would choose the higher paying one. And then, I would create and make outside of work. And since I have that much higher wage, it could be a jump start on my own business.
Yeah, I guess I've been lucky to be able to chose daily jobs in the past that basically gives me what you would create outside of work, except I got a fixed payment each month for doing something I really enjoyed. So I never had the need to do that stuff outside of work to derive enjoyment of most of my time, which I guess is my top priority and been most of my life.
That's reasonable, but signalling this to a startup during hiring is going to be a negative. There are three kinds of capital they get to play with, cash, equity, and culture. Cash is the least pliable of the three to them, equity the least liquid, but culture is actually something they can control.
If you have a team full of people who are just there for the paycheck, the only thing that will keep them there is increasing the paycheck. Which startups can't do in a crunch.
Culture is smoke and mirrors. When investors say frog, founders jump and guess who gets the cultural shaft then.
Better to be thinking in transactional terms from the get go especially in early startups, where majority of total comp is an illiquid call option.
I mean it's not smoke and mirrors, when I'm picking between jobs I'm strongly considering the people I'm spending 35-40% of my waking hours interacting with. If I cared solely about maximizing personal returns I wouldn't work for startups.
> but if I was choosing between two jobs where one was utterly boring but paid 3 times more than a fascinating job, I'd take the latter in a heartbeat.
I chose my current job against competing offers because it was a good thing for the world, but I would not have taken significantly reduced pay for it. Let me tell you why: nobody that insists on paying you below market rates is going to treat you right. Some of the worst professional interactions I've ever dealt with involved high ranking individuals at nonprofits. These were orgs may have had genuinely good missions, but also paid rank and file employees quite poorly. On the other side, the 3x above market rate job is just a fantasy. I could believe it if you were talking about a 20% bonus.
I hear you, and I've never solely chased money either. But, we unfortunately live in world driven by money, and if I'm going to pour myself into work I want to be compensated appropriately. I also have a huge issue with feeling like I'm getting taken advantage of. So, what I have done is try to find jobs where those things align somewhat.
I think we live in a better world when the primary motivator for how we spend half of most of our working day isn't doing what's necessary to get strips of paper for survival.
That is, the work itself should be interesting and fulfilling.
Yes, we need money, but when the work relationship approaches being purely transactional the whole thing is demeaning for everyone and less effective.
Boy, it takes a lot of luck and skill and privilege to be able to shop for (or offer) work like this, and money is still important.
If I wasn't paid, I wouldn't work for them.
While this is true for most of the cases, it's not the case always. There are definitely jobs I'd work for less than I make at the job now and there are jobs which are considered "dream jobs", but ultimately there needs to be a solid base to make you and your family feel comfortable if you're spending at least 1/3 of your day on.
They know it's a lie but still need to say it.
I used to have a manager that gave me this line. He left this company to join another one, had multiple offers, and told me he accepted the highest one because "it's all about the money".
They know, we know, everybody knows, but that's not the playbook.
> There was discussion about how they thought candidates asking about compensation to be something of a negative signal
It's not an unusual way of thinking, but every time I see it, it seems bizarre to me. If the candidate was to propose any project once hired, I'm sure these folks would want them to think about costs and benefits.
This policy selects either for people unable to reflect on their life with the same wit they apply to work; or people who will front about their motivations. Both seem like poor outcomes.
> The old Oxide compensation discussions were interesting. There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically.
I can't say anything of Oxide because I don't really know the people involved other than reading their writing.
This is an attitude a lot of industry veterans have. Most of them were in software pre-2015 and have long made their money and paid their debts. They saw the haydays of job hopping, massive salaries, massive equity, and "rockstars" (that never were). The people I know that joined software post-2015 and onward are not in the same financial position.
At this point I've accepted most equity comp in a non public company is worth less than toilet paper.
When it comes time to actually cash out or you notice some animals are more equal than others, the excuses start. The drama happens. No one knows what the equity is worth. If you try and advocate for yourself you'll be told the equity is actually worthless.
From here the 200k salary seems fair. Just don't live in SF. Chicago , Philly, Pittsburgh, Cleveland, quite a few cities are perfectly livable on 200k.
In fact, in any of the above metros you will be living nice.
Anyone who doesn’t ask about compensation (at least at a later point in the interview process) would be a red flag to me.
Most valuable people know they’re valuable, and do (and should!) negotiate compensation.
I am not sure I fully agree with the characterization above (that asking about salary is a 'red flag' in our process), but if I had to try and steelman it: we prominently put
> Everyone at Oxide makes $207,264 USD, regardless of location. (Some sales positions have a lower base salary and contain a commission component.)
On our applications page (see it here: https://oxide.computer/careers/sw-control-plane)
It's also a pretty well known aspect of the company. Combine this with the fact that our hiring process is different, where interviews are the very last thing before possibly being hired, and someone who has missed this fact could come across as having not done some very basic research about the company that they're applying to.
To be clear, I still think calling it "a red flag" is a stretch. I fully agree with you in a general sense, for places that are willing to negotiate compensation in the first place, but we make it very clear up front that we do not.
> (that asking about salary is a 'red flag' in our process)
> To be clear, I still think calling it "a red flag" is a stretch.
In my comment above I did not call it a "red flag". My specific wording was "somewhat of a negative signal". That seems consistent with what you said about judging someone for asking about a well-known aspect of the company because it signals they haven't done enough research.
It also dodges the question that keeps getting asked: Does everyone receive the same equity compensation as well? As far as I can tell, that question is not answered on your website. Asking it seems like fair game.
> In my comment above I did not call it a "red flag".
The person I was responding did, you are right that you did not.
> It also dodges the question that keeps getting asked:
This question gets asked in every thread about this, nobody is trying to dodge anything. The equity portion is variable, and the salary is identical. We do not do bonuses.
I think in general, at many companies a candidate asking is a red flag. Also for many candidates a company not wanting to discuss it until the offer is a red flag.
It seems that specifically your open disclosure very up front bypasses at least most of this from both sides.
FWIW, my last job I didn't negotiate. Company I really wanted to work for. Wanted to close the deal. So I didn't gum up the works with salary negotiation and it ended up being very good for me in a lot of ways.
Red flagging a candidate for not asking about compensation during the interview is not a good practice. This is an example of penalizing people for not following a specific script or candidate archetype you have in mind instead of judging them by their skills and abilities.
> There was discussion about how they thought candidates asking about compensation to be something of a negative signal because they wanted people who weren’t in it for the money, basically.
I think companies avoiding discussions about compensation is a negative signal, because they’re only in it for my labor.
> Does everyone at Oxide have the same equity grant?
> equity compensates for risk – and in a startup, risk reduces over time: the first employee takes much more risk than the hundredth.
I think that paragraph answers the question pretty clearly. As an Oxide employee you will get equity. It will generally be less than the people that came before you, but more than people that come after you. So it's obviously not the same as everyone else
It doesn’t answer it clearly at all.
Do two people hired at the same time get the same equity? Or is there room for one candidate to get more equity due to their experience or simply because they negotiated more?
Obviously early employees get more equity. The question is whether or not there’s room for negotiation. They heavily imply that everyone is paid the same, but all of those claims are about base salary.
"total comp" (salary+equity) is really hard to quantify for a private company. In order to qualify as ISOs, the stock options need to be priced at the Fair Market Value (FMV), which makes them essentially worth ~$0 on paper on the day they are granted. In order to value them differently, you need to guess if/how the company will increase in value in the future. If the gains were guaranteed, then that should be factored into the current FMV, so options always have significant uncertainty.
This is unlike an RSU from a public company, where you can sell the value of your shares as they vest and add that to your income with minor risk of price volatility.
Exactly right. It doesn’t answer the question. It does some cartwheels to avoid to the reality: despite their philosophies and soap boxing on compensation, equity is a form of compensation and their equity is no doubt distributed very, very differently.
It’s intellectually dishonest. I look forward though to their “in depth” follow up post.
They have since updated it slightly
> Since originally writing this blog entry in 2021, we have increased our salary a few times, and it now stands at $207,264. We have also added some sales positions that have variable compensation, consisting of a lower base salary and a commission component.
so basically everything converges to having specific reasons to have different salary schemes.
One exception, for sales. I don't see how they could have done it differently.
We have a "one rate per level" rule. The rates are published, and so are the definitions of the levels, and everyone's level (i.e. indirectly you can know everyone's salary)
Worked great, untill we started to look for sales. Doesn't work. They only know incentive-based schemes.
So now they have an incentive-based scheme just for the sales, which is (essentially) budgetted from their stock-option package (that everyone gets). I.e. they benefit from growth a little bit earlier and directer.
If we hadn't done that, we wouldn't have a sales department.
I argued this in the previous discussion about oxide’s compensation structure: I disagree that sales must be commission based. Yes, finding sales staff that are willing to work for a salary and equity shrinks the pool but the same is true of engineers willing to work for a flat rate across the company.
Sales might seem mystical and magical to engineers but it isn’t. A small company with a small sales team can absolutely work without commission. Yes, it is harder, but it is not impossible. The carve out for sales undermines the ideas behind a flat salary structure. Just because we can measure a sales person’s contributions in dollar amounts does not mean we must measure it in dollar amounts. Sales is as much about the partnerships between sales people and product/engineering, why aren’t all the people who work on a deal getting commission?
I’d go as far as to argue that oxide is in the perfect position as a big-ticket long-cycle business to abandon traditional sales commission structures. They take on all the negatives (sales people overselling to get commission) with no benefits. There are other ideas. Company wide bonus based on sales made during the year?
> Just because we can measure a sales person’s contributions in dollar amounts does not mean we must measure it in dollar amounts.
I don't even know if we can.
Yes, you can measure the number of deals signed they called dibs on. But:
1. You don't know if the salesperson earned it, or the whole product. There's a baseline demand driven by the whole company. This is the whole old argument that nobody can prove that ads work; you just can't pinpoint the purchase decision to exposure to an ad. So yeah I guess you can make your salespeople compete against each other and reward the one who stochastically floats to the top while punishing others. Sounds like such a fun workplace, I thought everyone agreed Microsoft's rank system sucked.
2. Several times I have witnessed salespeople selling non-existent, non-planned, functionality and forcing the rest of the company into crunch mode to not have a major client semi-publicly end the contract early. You're often just rewarding the biggest liar while everyone else has to cover up for their shit. Once again, sounds like such a fun workplace.
It comes down to, competitive sales is a cancer, and you're choosing to have it.
> Just because we can measure a sales person’s contributions in dollar amounts does not mean we must measure it in dollar amounts.
This is the fairest form of compensation. It's unfortunate that engineering contribution cannot be measured the same way. If we could engineers would all be getting a nice pay hike.
It's still just a market. You've got to make offers that people will accept. It's mostly silly to try to come up with some objective theory of value, except in the context of what potential employees will consider to be fair, which is right back at "you've got to make offers that people will accept."
Basing compensation on supposedly objective things like "the dollar amount a sales person brought in" might be important to a given pool of potential employees, but resist the temptation to think of it as objectively determining the value of the employee's work. Remember that all you're doing is making offers that people will accept.
I agree. But if employees know their objective value I believe it will change what they will consider an acceptable offer.
Only if the salesperson also implements & supports the things they sell. Selling false promises of something the rest of the organization has to fulfill is not fair -- but that's the way to the biggest commission!
And many would be getting let go because they weren't meeting some number.
Companies already do layoffs without commissions. They are always optimising to reduce salaries and increase profit margins. And its not "some number". Its the amount of $ brought in. Thats what companies care about.
Yes, but sales reps have very specific targets and sales managers have no problem routinely letting people go if they miss those targets. It really is a somewhat different situation from engineering--although projects, for example, certainly get canceled and teams let go. There's a more direct correlation to quarterly revenue/margin input in the case of sales.
But its a more transparent system. Right now no-one has any idea if they get laid off or if they are being underpaid. I think the overall compensation would likely go up if you are good at your job.
> One exception, for sales.
either you have a rule or you dont. the moment you start to do exceptions this will call for more down the road.
there is tangible reason for sales to be commision based. you could say its just like another regular position. An engineer can also be a sales multiplier if they improve a product yet they dont get a commission. Same thing. Having boots of the field is no justification.
Why is sales special? Why not just have some performance bar for that, just like all the other positions?
Because good sales staff make a ton of money through a ton of sales. Any other incentive structure is unlikely to attract high performers.
How is this not exactly the same in engineering though? Performance reviews at top tech companies are pretty much designed to identify the super high performers and shove massive bonuses and equity grants their way.
And those equity grants are effectively as good as cash, since they are publicly tradable stock.
It's pretty fundamental to the personality of people in sales to be driven by getting the sale and getting compensated based on the deal size. If you remove that carrot, it just doesn't work. Some sales people will make millions, some will make nothing.
> If you remove that carrot, it just doesn't work
How come it works for basically every other job on this planet? Developers aren't paid per feature implemented/bug fixed, and we still do those things, how come sales people are unable to do things for a fixed monthly salary?
>, how come sales people are unable to do things for a fixed monthly salary?
You have to separate out 2 different ideas of the "theoretical idealized salesperson that works for fixed salary" -vs- "real-world salesperson that works for variable commissions".
The businesses that have attempted to pay fixed salaries for salespeople end up attracting incompetent salespeople who can't sell. They become a negative cost on the company's payroll because they can't bring in any revenue. In contrast, the high-performance salespeople (the "rainmakers") are attracted to the variable high-commission, because they know they have the hard-to-find skills to actually sell and bring in the money. If a salesperson has the skills to get a customer to sign a contract and pay money, they have the leverage to get a percentage of that.
Developers, db sysadmins, tech support staff, etc are not in situations to directly influence and shake the hand of a new potential customer and convince them to write a check.
Sales works that way in every industry.
A top salesman can make more than the CEO from commission. Many top salespeople have a zero base salary.
The pressure is pretty crazy, though. I’m not cut out for that kind of thing.
I do know that (I myself also worked in sales for a short stint, unrelated to software though), what I don't understand how these magical "sales people" apparently can't work for a fixed salary when literally everyone else can. Apparently the rest of us can do high quality work without being paid for each feature/bug fixed, yet these individuals cannot?
> Many top salespeople have a zero base salary.
Hmm, probably true in some places, but here (Spain) that wouldn't even be legal. When I worked in sales we had minimum wage + commission, but I'm sure the salary would be 0 if they were allowed to set it up like that.
I have a few [wealthy] salespeople friends. Most have “commission-only” (0 base) jobs.
Many jobs will start you with a base for a few months, while you build commissions, but they stop it, after a while.
They can also get fired at the drop of a hat. Not much job security.
Sales are easy to convert to incentives. Just take a cut of the sale. It’s not so easy to calculate value from other jobs.
> Hmm, probably true in some places, but here (Spain) that wouldn't even be legal. When I worked in sales we had minimum wage + commission, but I'm sure the salary would be 0 if they were allowed to set it up like that.
In the United States having "zero base bay" is hypothetical. If a full-time employee had no commission payouts they'd be compensated minimum wage as necessary to comply with laws.
Sales jobs often come with a warm-up period with either a higher base salary or they get paid part of their commission target for a number of months regardless of how many sales they make.
> what I don't understand how these magical "sales people" apparently can't work for a fixed salary when literally everyone else can
It isn't that "sales people"[0] can't work for a fixed salary. The good ones just won't because they can find another employer that will pay commission, and they know they will make more with commission than without.
Employers will pay commission because that's how you attract the best sales people, and the best sales people are worth orders of magnitude more to their business than average sales people. Despite how much they earn, in general and compared to their average peers, the best sales people don't cost orders of magnitude more (5x is a more typical spread in tech sales.)
The advantage of 100% commission -- where it is legal -- is pretty obvious from the employer's view point. The company only pays for production. These sales people are commonly (but not always) independent contractors. The benefit for the sales person is a little less obvious, but, generally, they have more autonomy, a simpler comp plan without any caps, and earn more per dollar sold than they would on a base + commission plan.
0 - whether magic or not
> the rest of us can do high quality work without being paid for each feature/bug fixed, yet these individuals cannot
Yes they can not. It is not high quality work but high quality results for sales guy. Developer work is complete once service deployed. It wouldn't be developer failure if user volume doesn't reach x thousands per day on their web service. It would definitely be salesman failure sales does not reach x dollars in certain duration.
Developer equivalent of sales would be to say "I have distributed x sales brochures and call x number of clients this month. My job is done."
> How come it works for basically every other job on this planet?
In sales it is uniquely easy to identify your contribution since it's literally measured in dollars coming in.
There's no way to quantify that for feature implemented/bug fixed, it would devolve into endless politics.
> how come sales people are unable to do things for a fixed monthly salary?
Sure they are able, but no good sales person is interested. Presumably you'd want to hire the good ones. You can certainly staff a mediocre sales team on a fixed salary, they just won't do much selling.
> In sales it is uniquely easy to identify your contribution since it's literally measured in dollars coming in.
This logic about stops working as soon as you sell something that is not some immediately exchanged mass produced commoditized gadget. Sales people on commission have tons of incentive to saddle the company with demands and imaginary product features they can't actually deliver on and be long gone before the bill comes due.
Sales is unique because the monetary benefit to the company is mostly objective: If someone closes a $10 million sales contract, that becomes $10 million in revenue.
If a team of developers work together to fix a bug, how would you calculate the revenue value of the bug and how would you distribute that to the team that solved it? Technically the value of a bug is negative because it costs the company, so do you subtract that from the pay of the engineers he worked on it? If 5 people implement a feature that uses a library developed by 5 other people, which was built on the platform team's infrastructure, how do you divide up the commission? It doesn't work.
A $10 million sales contract is objectively $10 million in revenue, sure, but it's silly to attribute that entirely to the sales person just like it would be silly to attribute it entirely to the engineers that built the product or the marketing team that bought billboards.
Freelancers and consultants do absolutely have the option to get paid by the feature. It’s exceedingly rare for internal employees or contract employees though. That means there’s no market competition based on it yet. Some places do have bonuses or even profit sharing. Some senior ICs and many managers across the industry can get equity through either grants or options.
Sales professionals have a lot of different places they can sell things. The market-rate compensation for sales includes commissions. So to get the best sales people, you want something easy and exciting to sell with a good commission structure tied to the sales.
It's not that they're unable to; it's that the field attracts people who are financially motivated and other companies have compensation structures that reward personal performance.
Top salespeople generally won't work for a fixed salary because they want to make as much as they can, and the way they do that is by having as much of their compensation tied to personal performance as possible.
I personally think more engineers/developers should think the same way, but it's also much harder to directly tie job performance to compensation when contributing to a product.
> because they want to make as much as they can
But that's the same no matter if you work in sales, customer support or many other roles, a lot of people just care about the money with little regards to anything else, yet the sales department are the only ones who must have commission?
> But that's the same no matter if you work in sales, customer support or many other roles, a lot of people just care about the money with little regards to anything else,
It's actually not the same for many roles. See the comments from people in this thread alone who scoff at the notion of maximizing compensation. I don't get it personally, but it's not an uncommon thought.
> yet the sales department are the only ones who must have commission?
I think there's a very high likelihood that a salesperson is primarily driven by compensation, and good salespeople will already be working in a commission-driven compensation model elsewhere.
Why would a top salesperson at Dell, HPE, Oracle, or wherever else a hardware salesperson comes from move to Oxide to take less money and completely decouple their compensation from their performance?
Even if your intent is to maximize income in customer support, you often don’t have the market option to do that. I have seen (and even worked at) places where chat support, phone support, and support administrators have a quota of chats, calls, or ticket responses and make bonuses based on how much they exceed their numbers. Unfortunately sometimes that results in people updating tickets several times an hour saying things like “we’re still looking into this and haven’t forgotten you” without actually looking into anything.
One place I worked I tried to move the quota system more towards being the final response on a resolve issue, but upper management didn’t want to ever judge whether an issue was resolved even when the customer said they were happy. They did incorporate an NPS query for every interaction, though, and a multiplier against the volume-based quota. Unfortunately that favored people who were good BS artists when lying to the customer about looking into things.
The fallout from the above paragraph was that quality, caring staff would get punished for actually solving customer problems.
Almost every other role is at least one level removed from putting cash in the company's account, which is what leads to the shenanigans you described with metrics that are a poor proxy for revenue generation (and/or are too easy to game).
Because sales are quantifiable and directly mapped to performance.
To get that kind of proportional payback in engineering you'd need very clear financial objectives for a project. I could see that happening in optimization scenarios where consultants are brought in and get paid for whatever they can trim from operational costs.
In fact I’ve seen both tech and manufacturing efficiency consultants whose quotes include $x up front plus monitoring and reporting that shows the efficiency gains. Then rather than taking a closing fixed payment, they get a percentage of the savings to the client over the first six or twelve months.
The key difference between everyone else in a company and Sales is that Sales is where the money comes in, directly. It's approximately trivial to point at a sale, see how much money it made, and then who made the sale. So commission is a natural compensation structure for salespeople. For everyone else at a company, their individual contribution to the company making money is a lot more diffuse, and any metric you might be tempted to try and put a commission on is at risk of being gamed. Whereas "how much $$$/worth of stuff did we sell" is pretty much THE metric by which we judge a business as a whole.
If you’re asking this question then you have really no idea how the sales engine works at a company. It’s inherently incentive driven.
Because sales people are used to work on incentives, including going over and getting rewarded for it.
If they have a fixed salary with a high objective to "make it" (e.g. if you sell less than $X, you get fired), lots of sales folks will skip on it because they can't go over, and most probably prefer to have a quarter or two or year at e.g. 70% salary while working on longer term deals, rather than losing their job for not being good enough within that arbitrary time period. And going over their quota can be wildly lucrative depending on the terms.
FWIW, it seems like nowhere is this truer than SF/SV.
Outside the bubble, it isn't always the case, or the structure can be a bit different, but salespeople in the Valley (as it were) are a different breed.
Ask a local real estate agent or real estate broker how much base pay they make. Or, heck, a car salesperson or an ad salesperson at your local TV or radio station. It’s all commission-based. In some of these fields in the US the norm is 100% commission-based with no base pay. In others someone might make one to three times minimum wage, but will end up being some of the highest-paid people in the company based on commissions of anywhere from 1% to 50%, depending on the industry.
No, it's just how sales works, it's almost always on commission.
Tesla salespeople do not work on commission. Also you can align incentives through stock grants which appreciate by you selling more.
> Tesla salespeople do not work on commission
I work with tech salespeople with a variety of former employers as tech sales people, and I've never heard of anyone having worked without a commission. I'm vaguely in tech sales myself (solutions architect) and I'm on commission too, and so is everyone who joins our division from similar employment (solutions engineers/architect, or even customer success folks).
While I agree it's the norm, clearly Tesla has gone a different route and it's possible to do so.
Sales has been commission-based everywhere I’ve worked, including companies based in other countries.
Commission based sales was definitely not a Silicon Valley invention.
Sales culture is heavily incentive and performance based.
It sucks but they like to think that if they work harder they'll get paid more.
It's not universal but B2B sales in particular have evolved to incentive/performance compensation to a large degree. Wasn't always the case but (most?) of those companies aren't in business any longer. Also extends to the sales hiring process. Not that companies don't look at track records but it's also the case that sales managers don't have any issue firing people who don't meet their numbers.
Out of curiosity, how would you imagine that working out, technically and specifically? You start out with founders having equity, and maybe some early investors. Then you employ some people and... what happens exactly? Are new shares created, is everyone who was already onboard slowly diluted, what happens when someone joins and leaves? Etc, I don't think there is an approach to this where all sides would be happy, so I'm interested to hear your thoughts.
Not your parent, and not suggesting in any way that Oxide is one, but the key term you want to search to learn more about that kind of arrangement is "worker owned cooperatives." There's a lot of variety in how it's handled.
This has been the biggest red flag to me about oxide since these blogs came out. You don’t join a startup for salary and every startup I’ve been at that emphasizes fair salaries is doing that to intentionally discourage people from pushing on total compensation.
Employees apart from the very early ones are likely really getting fucked on equity because oxide explicitly treats candidates asking about equity as a red flag.
Companies structured like this end up turning into a combination of rich people who don’t need money and are just interested in the problem space and then mediocre people who can’t get a better offer. (If you are good, remote work with high TC is definitely available.)
Unless Oxide is giving out huge equity bonuses for good perf (which would make their comp post hypocritical), it’s going to have a continuous talent decline as it grows. There aren’t enough independently wealthy high performers interested in what Oxide is doing to sustain a talent pool.
> (If you are good, remote work with high TC is definitely available.)
But how good do you have to be at things other than the work itself (edit: and how lucky as well) to land one of those jobs with higher compensation? For someone outside of high-cost-of-living tech hubs, Oxide's fixed salary could be life-changing all by itself, even with minimal equity. The fact that they take that deal doesn't necessarily mean that they're mediocre.
> For someone outside of high-cost-of-living tech hubs, Oxide's fixed salary could be life-changing all by itself,
I’ve spent a lot of time looking at compensation tables and reference data. $200K is definitely good salary, but I wouldn’t call it “life changing” relative to what someone qualified to work at Oxide could earn in an average non-tech hub metro area.
The type of candidate who qualifies to work at Oxide has numerous options for high paying remote work, and probably well paying local work too.
That would be even more outlandish, I don't think that's possible.
It depends, there are certainly founders who come from sufficient money that the value of founding a unicorn is not material to their Net worth.
However I’d also suggest that the concentration of tech in the last decade is also partly due to startups chronically stiffing their employees on equity. The difference in compensation potential naturally forces a talent split where talent joins larger and much better compensated firms.
Of course not, what's the purpose of asking such as silly question?
Are you being snarky and suggesting that employees deserve the same upside that founders deserve?
> Of course not, what's the purpose of asking such as silly question?
How is it silly if they already do that for salaries? Co-ops with equal ownership isn't unheard of, and isn't silly at all.
Yeah they aren't unheard of at all, co-ops have had slow but steady growth in the market for some decades now.
I’m not sure I follow. In what market(s)?
Agriculture has a lot, utility services including fiber internet, electrical, and natural gas, many credit unions, there are a few retail stores like REI, mutual insurances, a number of pharmacies and homecare and other health services are co-op, many other worker co-ops, and there are a decent number of housing co-ops. Other than maybe a few rural fiber co-ops most are not super fast growing businesses, and they lack attention seeking rich owners, so they don't get a lot of attention, but co-ops as a whole has steadily gained more market presence nearly across the board over the decades.
One famous example is Igalia, doing Open Source consulting for various companies, including Google, Apple, and others.
They're a worker owned co-op and have grown very nicely over the years.
Igalia are very cool! Not sure if it’s so easy to reproduce their success (if I’m wrong on that point, that’s only good I guess).
> Are you being snarky and suggesting that employees deserve the same upside that founders deserve
The founders are excluded from the employee compensation discussion. They own the company because they founded it. Nobody thinks they just put the equity into a structure that nobody owns.
The question is whether all employees are compensated equally, which is a very important detail. Giving everyone the same salary is very different than giving everyone the same total compensation.
Salary and equity have nothing to do with what you "deserve", only what you're able to negotiate.
For early/mid stage startups - this is an awful position to take. These orgs are heavily influenced by who they hire - what you pay defines your incentive structure.
Does the world class engineer or business development lead just take it easy and travel around after they join?
Does the new manager push the team and business forward or prioritize stability?
Do engineers spend their time on reactors and impressive sounding projects or figuring out what customers need?
Do people feel lucky to have a seat in the org or do they spend their time complaining and looking for the exits?
Money isn't the only lever, but its a strong one - startups will never compete with established firms on cash outlays.
People are paid salary and awarded equity based on supply/demand for labor, the marginal product of that labor, and the amount of risk engineers are willing to accept by joining a startup. It's an economic transaction, the same as buying office equipment and signing contracts for cloud resources. Trying to imbue mysticism into it is just asking to be lied to by your employees
There is no mysticism in incentive structures. My point was rather that if you provide strictly below market compensation (as most startup equity is positioned these days). You are likely to get below average talent, or below average results from poor incentives.
Would love to engage in a discussion with you on this. How would you describe "deserve" in the sense of compensation? I agree with your premise that what you get is ultimately bound by the ceiling of the payer's generosity and your ability to negotiate.
But what sorts of things input into the function of "deserve"?
Everyone deserves healthcare, a place to live, food to eat. Some people deserve to live happy lives and some people deserve to rot in prison. These are about your personal conduct and how much you contribute to society.
How much equity or salary you get in a company is a function of supply/demand and the marginal product of your labor. I would say there are probably fewer CEOs who can take a company from startup to unicorn status than there are really good founding engineers out there, so CEOs tend to get more equity in a company. Sometimes the founding engineer knows something that nobody else in the world does, so their equity reflects that. It's also a reflection of how much risk the engineer is willing to take on (they'll probably take a salary cut to be a founding engineer, and they also risk the company randomly running out of runway and finding themself suddenly unemployed).
But it has nothing to do with what you deserve. Maybe if the CEO/President is a sentimental type, he'll award you equity based on how much he feels you deserve but ultimately it's about supply and demand.
If a CEO puts in 90 hours a week at a tobacco company while his engineers put in 20 hours a week, does he deserve lots of money (and therefore a more comfortable life) because he puts more effort into killing people? Or does he deserve every bad thing that happens to him because he decided to spend his limited time on this earth making it a worse place?
A lot of people with broken /s detector are replying to this comment.
It would seem like parent's something detector is off if they think the grand-parent's comment was snarky.
That isn’t a snarky position; early employees in high output orgs like this generally work just as much as founders do.
The founders aren’t really taking on that much more risk than the rest of the early team; it’s the VC’s money, not theirs.
I absolutely don’t agree with the idea that employees deserve the same upside as founders (because I think initiative and persistence against adversity/inertia is insanely rare and valuable and should be rewarded immensely), but it is not an insane proposition.
It’s especially popular among people who think the actual work output is more important than the leadership initiative. Both are obviously essential, and founders do both, while employees do only the first (or they’d be founders themselves).
>rewarded immensely
Let’s define this. Let’s say 1:25? 1:50? What ratio is appropriate?
We don’t need to define it; employees define it by who they choose to work for given the equity granted to them by the founders.
If they didn’t like the deal, they would become founders themselves, or choose a company that offers a better deal.
It turns out that leadership drive and the compulsion to bring something new into existence from scratch is actually quite rare.
Your figures seem to be roughly in line with what the employment market has settled on, although price discovery could be better (most employees don’t get to see the cap table during hiring negotiation, which, IMO, is wrong).
Pretty bullish, anyone who has tried to setup and manage their own compute knows the pain they're solving.
Plus I predict more companies will exit the cloud once they realize how thick the margins have become or want better guarantees over sovereignty.
I actually think that having more cloud providers might deflate a lot of the pricing. If you think about it, companies like Amazon buy server hardware and then rent it out by the vcpu (with throttling if they can get away with it) per month. Add memory and IO and you are looking at bills that pay for the server in mere months/weeks several tenants carving up all the hardware and each paying tens/hundreds per month.
There are of course benefits to using cloud based VMs and I use them as well. But you are paying a very steep premium for what is a pitiful amount of compute and memory. There's a lot of wiggle room for price decreases and the only thing preventing that is a lack of competition. There's a reason Amazon is so rich: nobody seems to challenge them on AWS pricing. There's value in having them do all the faffing about with hardware of course. That's why companies use them. I'm in GCP; but same principle. I don't want to have to worry about replacing hard disks in the middle of the night, deal with network routers that are misbehaving, cooling issues, etc. That's why I pay them the big bucks. But I'm well aware that it's not that great of a deal.
I used Hetzner a decade ago and paid something like 50 euros per month for a quad core xeon with a raid 1 disks, 32 GB, etc. Bare metal of course. But also, 50 euro. We had five of those. Forget about getting anything close to that with modern cloud providers for anything resembling a reasonable price. Your first monthly bill might actually add up to enough to buy your own hardware. Very tempting. They have beefed up their specs since then. You now get more for less. And they also do VMs now.
I knew it was bad but I didn’t realize just how bad the pricing spread can be until I started dealing with the GPU instances (8x A100 or H100 pods). Last I checked the on-demand pricing was $40/hr and the 1-year reserved instances were $25/hr. That’s over $200k/yr for the reserved instances so within two years I’d spend enough to buy my own 8x H100 pod (based on LambdaLabs pricing) plus enough to pay an engineer to babysit five pods at a time. It’s insane.
With on-demand pricing the pod would pay for itself (and the cost to manage it) within a year.
That's just hardware. If you need to build and maintain your own devops tooling it can balloon in complexity and cost real quick.
It would still likely be much cheaper to do everything in house, but you would be assuming a lot of risks and locking yourself in losing flexibility.
There is a reason people go with AWS over many competing cheaper cloud providers.
> There is a reason people go with AWS over many competing cheaper cloud providers.
Opportunity cost.
The evolutionary fitness landscape hasn't yet provided an escape hatch for paying this premium, but in time it will.
In my experience, companies seem to want to pay the cloud provider tax in order to avoid capacity planning. Sometimes it makes sense because it is hard to predict when something is going to take off. I have also worked at companies with very predictable growth paying insane amounts. I didn’t understand the logic, but they still were profitable and paid well so whatever.
At the (very) low end it's pretty easy to build your own "cloud" with a NAS, containers, and reverse proxies and tunnels to the outside world. And this will get you suprisingly far.
But at the high end, I think the market is litterally infinite. Every large company should want this, and want it now. Cloud providers are extremely expensive and, outside of the 1-tier where prices are really outrageous, they perform poorly and often offer little support.
This really feels like the future.
> At the (very) low end it's pretty easy to build your own "cloud" with a NAS, containers, and reverse proxies and tunnels to the outside world. And this will get you suprisingly far.
Anyone can throw together a bunch of parts and software to run Internet-facing services from a closet. That doesn't mean that you're safe from issues that Oxide aims to solve, especially at that small scale.
My homelab (which hosts my blog and a couple of other things) runs off a Topton N17 micro-ATX motherboard ordered on AliExpress, featuring an AMD Ryzen 7 7840HS. Yes, that's a mobile CPU shoehorned onto a desktop platform with a funky mounting bracket to take AM4/AM5 coolers.
Anyways, I wanted to run SmartOS on it, but this system is so janky that the Illumos kernel couldn't find any PCIe devices at all. After spending an afternoon reconfiguring PCIe bridges by hand with the kernel debugger in an attempt to troubleshoot PCIe initialization, I gave up and installed Proxmox.
Admittedly, as far as janky hardware this takes the cake, but the point stands. To paraphrase Bryan, buggy firmware is the sysadmin's worst enemy.
like us old assholes were saying when the cloud really started to take off: "this is nuts, it's just someone else's computer! and they're making a profit off of this service, meaning it's more expensive than what we were doing!"
Now a lot of the things that were done pre-cloud were done in bad ways, and I'm not saying that we were right about those things. Having APIs for provisioning and monitoring are far better than submitting a request to some queue and having your VM provisioned manually 1 week later by someone who gets a key detail wrong. APIs and granular permissions are how this should be done, and "the cloud" taught everyone that very early. But a lot of companies are really stuck in the cloud mindset now, and won't let go of it.
I think companies like Oxide and product lines like theirs are going to start becoming common. Microsoft, of course, completely fumbled the ball with Azure Stack, and I've never even heard of anyone deploying AWS Outpost, both for the same reason: the costs for these are absolutely insane for what they provide.
What most folks really want is their own infrastructure running their own stuff using APIs that are either written in-house or provided by some vendor. Oxide is betting that they can sell you a working scalable system for less money than it would take to hire a team to write the APIs that would allow a company to do the same with off-the-shelf hardware. I think that they're probably right about that.
> they're making a profit off of this service, meaning it's more expensive than what we were doing!
I hope you can see that this is a logical fallacy known as "zero-sum thinking". It is not only possible for a business to profit while lowering prices, it is universal throughout the economy. Tomato farmers make a profit selling tomatoes at a price much lower than the cost to grow tomatoes at home. Bakeries radically undercut the cost of home baking. It is obviously cheaper to buy motor fuel at the gas station than it would be to buy crude and refine it yourself.
The main reason people think their on-prem is cheaper than cloud is that they are bad at accounting.
https://arstechnica.com/information-technology/2024/10/basec...
> [2023] 37Signals expected to save $7 million over five years by buying more than $600,000 worth of Dell server gear and hosting its own apps.. [2024] update: it's more like $10 million (and, he told the BBC, more like $800,000 in gear). By squeezing more hardware into existing racks and power allowances.. transferring its 10 petabytes of S3 storage into a dual-DC Pure Storage flash array, 37Signals expects to save money, run faster, and have more storage available.
So, to dig into (and maybe stretch) one of your analogies a little bit:
> Tomato farmers make a profit selling tomatoes at a price much lower than the cost to grow tomatoes at home.
This is because at home, you have an elastic need for tomatoes. One week you need a few, the next week you need none, the next week you need a lot. In that case, yes, growing your own tomatoes would be very silly. (This part of the analogy works with on-demand instances.)
However, if you aren't a home, but you're a busy restaurant, you're not buying tomatoes from the grocery store. You have a regular, fairly fixed capacity, and so you go with a produce vendor who's able to serve that need at a decent price. Part of the reason you're able to get a cheaper price is through volume and due to the regular-ness of the business. (this part of the analogy works with reserved instances.)
Okay, so let's move from tomatoes to the actual building of a restaurant itself. Similar to a reserved instance, renting a building is a decent way to get started with less capital, and you have fairly consistent capacity requirements. But at some point, you realize what McDonalds did: owning the building and land beneath it ends up being a great deal at certain scale. Because that ends up being cheaper still, in the long run. This is closer to on-prem. (Okay at this point this analogy is getting pretty silly but it was fun to try and work through it...)
So, the trick is, for a lot of organizations, they could realize the benefits of owning their own hardware, but to get back to your original analogy, running your own hardware comes with its own set of costs that may make it not worth it. You have to have staff to operate everything, you have to manage all of the various supplier relationships, keep track of software licensing fees, etc etc etc. Even with all of this, as my sibling comment shows, often this can be cheaper than using the cloud.
But one way of looking at Oxide is, we are making it so that it's simpler to own your own hardware, thanks to all of the integration work we do. You don't have to manage a ton of vendor relationships, you have "one throat to choke," as they say. You don't have to keep track of software licensing fees, there are none. You don't need to build out your own software to put the whole thing together, we give it to you. Etc.
So yes, just like a household is best served by going to the supermarket, individuals aren't ever going to buy Oxide. But there's a lot more out there than just households. And larger organizations have fundamentally different needs than they do.
> The main reason people think their on-prem is cheaper than cloud is that they are bad at accounting.
I mean, there's also base accounting stuff that differs significantly between the two, like opex vs capex spend. Not that I'm an expert in that, mind you.
My comment wasn't really about Oxide, it's about the fallacy. A person can't succeed in life thinking they can do something cheaper on the sole basis that the other guy makes a profit. The much more likely explanation is that the other guy makes their profit by being much, much better than you.
To stretch the metaphor to a grotesque extent, I think Oxide stands in the middle between the home-grown and the industrial tomato. Your average corporate IT installation has the same economics as home-grown. Even if they have 1000 potted plants, they are still potted plants, and they are still $100 tomatoes. EC2 is a 5000-acre California tomato grower where the fields have been leveled using lasers and the fruits are harvested by robots driving themselves using on-board GPUs. Their tomatoes cost 5¢. An Oxide computer is like having a 1-acre kitchen garden where the tomatoes are in rows. These are more like $1 each. The economics are undoubtedly better.
> A person can't succeed in life thinking they can do something cheaper on the sole basis that the other guy makes a profit.
That's fair! Just like, pointing out that on-prem can make financial sense.
> To stretch the metaphor to a grotesque extent,
I like it, haha.
I don’t disagree that there are some fat margins in the cloud, but how is vendor lock-in any different here? Companies could end up paying fat margins to oxide too while still managing physical gear and plant.
Well, their servers are mostly hypervisors, so the interface is mostly any virtual machine.
You can "just" migrate by exporting or importing the vms.
You're literally ignoring that the Oxide management stack is very much custom and effectively vendor-locking the purchase to be maintained by them. They are not general purpose PC servers.
You can "just" migrate away from Oxide but that would mean throwing away the hardware you now own. That's the grandparent's point; if you're migrating out of a cloud to avoid the margins demanded by the cloud vendor, now you're at the mercy of whatever Oxide thinks your support contract is worth.
Sure, the convenience may be worth it, but watch how many companies are now struggling to get off of VMWare after Broadcom moves.
This reminds me of people complaining about github being closed source and moving to gitlab, or people obsessing over terraform to avoid cloud locking.
Sure you will have vendor locking at the periferies, but the core is what's important, the guest vms. The hypervisor is whatever. If you have 100 vms running on ec2, you have done a great job of designing portable software, don't obsess over the last 1%.
Having done web applications for oh…30 years now the key pain is network routing
That’s the only thing I can tell is useful about “the cloud”
I can build racks and servers easily, but the challenge is availability and getting past everyone’s firewalls
So the real win is any service that allows for instant DNS table updates and availability of DNS whitelisting.
This is why Google, Msft etc win in email because they have trusted endpoints
Alternative routes with self signed DKIM etc is more or less blocked by default forcing you onto a provider
We need more cloud flare tunnel and local hosting via commercial ISP routes and less new centralized data centers
+1 Insightful.
I've also been doing web-related work for a living for over 25y and yours is the most spot-on take I've seen in this discussion.
Agree. But those fat margins can get starting to be shared with CIOs, CTOs and other managers with purchasing power. IMO the constant hectoring at workplace about migration to cloud or cloud native crap is not coming from some deep technical principles. It is more of Do it before you get fired for non-compliance
I've run my own servers for over 20 years now, but I guess I don't understand the pain point. Can you elaborate? They write:
> Our system delivers all the hardware and software you need to run cloud…
I'm literally so old that I write programs that run on a server and never think about infrastructure.
> To "run cloud?"
I agree that's a bit awkwardly phrased, let me send in a patch for that.
> Does this mean treating your own servers like "serverless?" Does it mean running Kubernetes? Is this primarily for people who want to self-host LLMs?
Not exactly any of that. We let you treat an entire rack as a single pool of resources, spinning up virtual machines that our control plane manages for you. Think "VPS provider but you own it." There's an API, but if you want to see what our console looks like, you can poke around with a demo here: https://console-preview.oxide.computer/
> To "run cloud?" Does this mean treating your own servers like "serverless?" Does it mean running Kubernetes? Is this primarily for people who want to self-host LLMs?
Machine/system/service deployment via API.
See §Essential Characteristics
> On-demand self-service; Broad network access; Resource pooling; Rapid elasticity; Measured service
* https://en.wikipedia.org/wiki/Cloud_computing
None are generally applicable to standalone pizza boxes as managed individual (as opposed to being herded by (e.g.) OpenStack).
No GPU’s yet, so it’s not good for LLM’s. But they have a lot of funding now, so perhaps that will change?
Oxide, at least for an outsider, looks like a company that channels some of the spirit of early Sun Microsystems (I'm aware of the connections of course). I'm quite envious of those who work there - I hope the demands of big money don't crush any of that spirit.
Sadly when I look at their jobs posted I don't see much that would line up with my skillset, but I keep an eye on them just on the offchance.
Right?
And such a clear value statement.
If I could, I'd invest. Sure, they might fail, but they're shooting their shot, and to me it has a clear differentiator that would improve the market for most of their users (if not the incumbents).
I wish there was a way for small investors ($25-50k) to get in. AFAIK, the only thing we can do is wait for an IPO and hope we can get in at a reasonable price.
Meta: Oxide has talked about the designs of their cooling [1][2][3], so I'm curious to know if they ever start offering GPUs how they'd handle that.
Folks seem to be moving toward liquid cooling[4] either to the rack/chassis[5] or even to the chip[6].
[1] https://oxide.computer/blog/how-oxide-cuts-data-center-power...
[2] https://www.youtube.com/watch?v=4vVXClXVuzE
[3] https://www.youtube.com/shorts/hTJYY_Y1H9Q
[4] https://blogs.nvidia.com/blog/blackwell-platform-water-effic...
[5] https://datacentremagazine.com/data-centres/top-10-liquid-co...
Oxide's blogs talk about cooling _a lot_, apparently their stack runs very cool b/c they've reorganized the whole thing around efficiency, and written all the firmware to support that goal.
Skeptical of that. There's only so much you can do against the physics of moving electrons around at high speeds... "Bigger Fans" and "compute density" doesn't change that
Commodity hardware doesn't quite tend to operate at a compute vs efficiency Pareto frontier — there's a lot of wasted energy that we've been able to optimize with our vertical integration. (I work at Oxide.)
I don't have any particular knowledge about oxide's cooling, but think about how bloated and inefficient literally every part of the compute stack is from metal to seeing these words on a screen. If you imagine fixing every part of it to be efficient top to bottom, I think you'll agree that we're not even in the same galaxy as the physical limitations of moving electrons around at high speeds.
But the majority of heat is going to come from the CPU and this is a product to run arbitrary customer workloads.
If the customers leave these things idle, then oxide is going to shine. But a busy rack is going to be dominated by CPU heat.
According to Oxide Computer, they found that going from 20mm to 80mm fans dropped their chassis power usage (efficiency is to the cube of the radius): a rack full of 1U servers had 25% of its power going to the fans, and they were able to get down to 1.2%:
* https://www.youtube.com/watch?v=hTJYY_Y1H9Q
From their weblog post:
> Compared to a popular rackmount server vendor, Oxide is able to fill our specialized racks with 32 AMD Milan sleds and highly-available network switches using less than 15kW per rack, doubling the compute density in a typical data center. With just 16 of the alternative 1U servers and equivalent network switches, over 16kW of power is required per rack, leading to only 1,024 CPU cores vs Oxide’s 2,048.
* https://oxide.computer/blog/how-oxide-cuts-data-center-power...
20mm fans aren’t used in server cooling applications. You must be thinking of 40mm fans.
Going from 40mm fans to 80mm fans will not take energy usage from 25% to 1-2%. They must have taken an extreme example to compare against. What they’re doing is cool, but this is a marketing exaggeration targeted at people who aren’t familiar with the space.
Oxide also isn’t the only vendor using form factors other than 1U or focusing on high density configurations. Using DC power distribution is also an increasingly common technique.
To be honest, a lot of this feels like Apple-esque marketing where they show incredible performance improvements, but the baseline used is something arbitrary.
Our claim is not that just switching fans drops from 25% to 1-2%. We are claiming that the rack has very low energy usage, and we like to talk about the fans as one part of that reason because it's very visceral and easy to understand.
I think 1U was poorly optimized for scale, and thus bigger chassis in a rack could use bigger heatsinks and fans at lower speeds instead of small screamers.
This is not any different than the "blade" form factor that was popular in the 90s. Shared power and cooling that was not constrained by the height of a 1U rack chassis, with larger fans. Hell, even Supermicro has blade-style chassis with 80mm fans. This is not novel.
It's just plain old engineering, optimized to sell whole racks not individual servers or <=8U units, sprinkled with opinions about low-level firmware etc, with a bespoke OS and management stack.
Yes, we're primarily an engineering company, not a research organization.
It's also about what we don't have. We don't have a UEFI, for example, which means we don't have UEFI vulnerabilities.
Yeah and you're doing good work there. It just kinda annoys me when people go from "oh that's a cool company" into idolatry. 1U servers were always a poor form factor for modern day hot chips & drives. Breaking that mold has been done over and over and isn't something that should be treated as new.
Scaling from the 8U (that blades could already do in the 90s) to full rack as the unit of "slide unit in to connect" DC power and networking is way cooler than using 80mm fans.
Re UEFI: I feel like that part is less about UEFI itself and more about how you have very minimal third party firmware.
I'm pretty excited about openSIL and such in general. If only AMD could execute well in the world of software.
I can't speak to others' views, but having worked with large-scale bare-metal deployments at Meta, I personally admired Oxide for its clear product vision and rigorous first-principles approach (Rust is a real game-changer!), and applied to work here for that reason.
> It's just plain old engineering, optimized to sell whole racks not individual servers or <=8U units, sprinkled with opinions about low-level firmware etc, with a bespoke OS and management stack.
Yes, "just".
An F1 car is also just plain old engineering, optimized to get around the track quickly, sprinkled with opinions and with a niche bespoke drivetrain. Nothing to see here.
Their rack scale from-scratch redesign includes fans big enough that they've reportedly managed to cool CPU hardware that was actually designed for water-cooling, with no expectation for air cooling (though admittedly, they say they only achieved this just barely, and with a LOT of noise). That seems like something that's going to be objectively verifiable as a step up in efficiency.
I believe the fans are actually smaller. The rack is definitely quieter than other racks, but he says in the rear rack tour that it's quite hot. Check out these videos of it
[0] - https://www.youtube.com/watch?v=dHbgjB0RQ1s [1] - https://www.youtube.com/watch?v=lJmw9OICH-4
maybe you're not familiar with just how stupidly written most code is.
you're right that there are efficiency limits, but not once in my career have I ever seen anyone even attempt to write their code so that it is efficient to run, outside of gaming.
Oxide is doing some great things, but there’s only so much you can do with firmware tweaks. A CPU running any load at all is going to completely eclipse the power usage of everything else in the system.
Incremental improvements from things like more efficient fans and reducing the number of power conversions is great, but the power drawn by the CPUs or GPUs is on another level.
If after that you're not satiated with data center cooling talk, Jane Street's Signals and Threads just did an episode about their cooling infrastructure a few days ago:
https://signalsandthreads.com/the-thermodynamics-of-trading/
There is something so calming and pleasant about a well-structured thesis statement:
> Our thesis was that cloud computing was the future of all computing; that running on-premises would remain (or become!) strategically important for many; that the entire stack — hardware and software — needed to be rethought from first principles to serve this market; and that a large, durable, public company could be built by whomever pulled it off.
Very clear and logical, stating from their first principle world view what the result could be if they succeed.
I would say it's very clear and logical ... but is it really "from first principles"?
I thought that Oxide was based on OpenCompute, which is basically the rack designs that Facebook open sourced, after hiring some Google employees to build their custom data centers. This project started in 2011:
https://en.wikipedia.org/wiki/Open_Compute_Project
https://github.com/opencomputeproject
Google definitely rebuilt the stack from first principles -- the data center was basically a huge embedded system, from power to racks to CPUs/memory/disk/network to kernel to user space to cluster software. (And no, they did not use Kubernetes.)
I have no idea how active the OpenCompute project is -- is Facebook still the main contributor, and are they still releasing their new rack designs?
And I also wonder how much Oxide has diverged from it? At least on the hardware side. On the software side, I guess the Illumos-derived parts and Rust parts are completely different.
Well, when they say they did their own:
- board designs
- microcontroller OS
- platform enablement software
- host hypervisor
- switch
- integrated storage service
- control plane
Maybe they only got the mechanical and power stuff from OpenCompute? i.e. the parts that change more slowly
We didn't use anything from OCP. When we first started the company, we thought we might use the enclosure (and considered ourselves "OCP inspired"), but there ended up being little value in doing so (and there was a clear cost). And on the stuff that we really cared about (e.g., getting rid of the traditional BMC), we were completely at odds with OCP (where ASPEED BMCs abound!).
So in the end, even the mechanical and power didn't come from OCP. We clearly build on other components (we didn't build own rectifiers!), but we absolutely built the machine from first principles.
My read on the situation is that you copied OCP only inasmuch as OCP made some observations about physics and you made the same observations.
The most obvious change is that you guys use half-width, full length enclosures.
But I'm realizing now that I haven't looked at the OCP specs in a long, long time. I recall the common DC rail from some early Facebook papers on this topic, and I thought they were pretty similar to how you plug into power.
I just looked at the OCP power connector, and I would have lost any bet anyone was willing to make me about what they looked like. That's not at all where I thought we were. I think I understand now why you guys went to such pains getting the keyed connectors to work exactly right. Their power connectors look like something from a scifi movie, and not in a good way.
Ah OK, thanks for clarifying! I'm glad to see these kinds of machines being built, especially with so much open source software
How does the Oxide 48V architecture compare to what is known about the Google/OCP architecture? Does Oxide use single stage conversion, intermediate 12V buses, or ??
I'm not someone who works on this part of the product, but we talk a little about this stuff here:
* https://oxide.computer/blog/how-oxide-cuts-data-center-power...
* https://docs.oxide.computer/guides/introduction
I feel like we had a good Oxide and Friends on this too... https://oxide-and-friends.transistor.fm/episodes/bringing-up... has some info about our power setup.
Anyway, I barely know anything about this topic, but I think this answers your immediate question: we convert AC -> DC once at the rack level, and then use a bus bar to distribute that to each sled. Each sled also has a converter to convert that 54V down to 12V for its own bus within each sled.
Probably the sweet spot, since you can get to market fast with known 12V designs, but still enjoy the possibility of later announcing you've made the sled even more efficient by getting rid of the intermediate voltage!
At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.
The thermal dissipation is a function of the current squared. The heat in the conductor is a function of the size of the conductor and the surface area for heat dissipation. So these high current common rail systems you can sometimes see in youtube videos for power distribution, have great honking bars of copper in them. And in most of the videos I've seen, the video is about someone screwing one of these up, damaging the bar, and now the electrician has to wait for a new one to arrive, because they are shipped from far away and they are expensive per pound, and the dumb things weigh many pounds.
So you don't actually want the power to be at 12V for very long in a power dense rack. Their spec sheet says that each rack can pull 15KW. And that's wired for 208 or 3-phase power. That's 10 hair dryers of power per rack, so yeah maybe you shouldn't step it down until the last responsible moment.
Do any parts of the rack run at the full 54V? That would make for some very nice cooling fans.
In Oxide's design, we do have a 54V DC busbar so that's what the rectifiers put out, and runs vertically up and down the back of the rack. The power connection into each of the cubbies for the sleds, and the power into the sidecar switches connect to this bus bar at 54V. Each of these assemblies has an intermediate bus converter IBC that does the 54->12V conversion on board and 12 and other lower rails are used for the various supplies required.
We do run the 54V to our fans (in both sleds and switches) without additional DC-DC conversion as those can be fairly power hungry and we can buy reliable fans that are rated for this voltage.
Our sleds actually don't connect directly to the bus bar to help mitigate some of the "oops" factor as they're going to be potentially mated and unmated buy customers as they reconfigure, upgrade or support things. The sled cubbies are wired to the bus-bar and support hot insertion of the sleds. And yes, while possible, an in-field replacement of the bus bar wouldn't be fun, but in our design it's a big copper bar hidden away so the risk of damage or dropping stuff into is minimal.
So our 12V IBC design gets us into more normal range for commodity point-of-load supplies, and balances the losses due to higher current at 12V vs the complexity of dealing with 54V all over the boards. For the AMD parts, we also have to have supplies that deal with SVI2 or SVI3 where the part itself can adjust its voltage at run-time for efficiency. These are pretty complicated devices (like the RAA228218) that we're happy to not have to design ourselves and they have expected operating envelopes for their supply-side rails that don't work at 54V.
It's certainly the current mainstream style to have an intermediate voltage rail of 12V or more. But this OCP talk from a few years ago was interesting, showing a prototype direct 48V-1V conversion with high efficiency.
Yes it certainly can be done, but there's a cost and design complexity with doing that too. I did a quick count of gimlet (our server sled's) power rails and got to over 26 different power domains, and I probably missed a few in my quick scan! It's unclear to me if the efficiency gains from re-doing these with something more exotic to go from 54V would make enough of a difference to justify doing so, and we'd still end up with some stuff like the SVI2/3 controllers needing an intermediate rail (or have to go design those ourselves too) and some analog rails needing LDOs for noise rejection reasons etc. As mentioned before, creepage and clearance at higher voltages cascades into layout complexity and pain if you have to run it everywhere on the board: but for the same reasons we're talking about this, we can't very well do a 54V:1V conversion in the back of the sled and run it all the way to the front- losses, noise etc.
As with all things engineering is a series of tradeoffs and right now, an IBC from 54V to 12V has been a reasonable design point for us.
We initially talked about OpenCompute, yeah, but as far as I know we ended up moving away from that years ago.
>public company
Why does it have to be a public company though?
It is either that or acquisition, and we would prefer an IPO.
Speaking from the experience of 8 acquisitions/IPOs, your compensation structure will not survive public ownership. It could survive a PE acquisition, but definitely not public ownership (in my experience). I would recommend petitioning for a dual-class share structure to protect founding leadership.
We agree that we don't know what the future holds, and what makes sense for early stage companies may not make sense for later ones. When anything stops working, we'll make changes. You can see this already with the secondary structure for sales. We have a while before we need to worry about that, though :)
Because VCs need to have a way to sell their shares within the (limited) lifetime of their fund.
Good for them! I used to walk my dog past their (office? warehouse?) in Emeryville, and when the weather was warm they'd have the doors open and the giant server stacks just sitting there, looking awesome. I guess it's not really a concern that someone will steal something that looks like it'd take a forklift to move.
The ultimate aspirational homelab setup, ngl.
Congrats to Oxide on this milestone! I’ve been following their progress since discovering them in COVID, and would love to see them shake up what’s presently a stagnant marketplace with their product line. The idea of deploying a rack of kit on-prem that’s tightly integrated instead of wrangling multiple vendors of discrete components has a strong appeal, and while the proprietary hardware stuff did initially give me pause, their commitment to building atop Open Source quelled any lingering doubts I had.
Would love to see their growth result in more versatile options, like quarter-rack or industrial deployments someday. In the meantime, congrats on the successful fundraising!
In the era of anything even remotely AI-adjacent raising ridiculous amount of money - and proving to vaporware - this is one raise that makes me happy. We need more of the Bryan mindset (and less of Larry) to ensure that when the AI-Enabled future shows up we have some semblance of a fair society that works for all. https://www.youtube.com/watch?v=-zRN7XLCRhc&t=2047s
Aside from the actual product, On the Metal / Oxide and Friends are really great podcasts that manage to make programming topics entertaining and educational. Bryan Cantrill is wildly entertaining and knowledgeable at the same time. His co-hosts and guests are great, too, and I attribute a lot of that to feeding off of his energy and storytelling. Highly recommend, especially for Rust folks.
All the best! I personally came to know Oxide for their cool RFD culture. It's worth a read:
https://rfd.shared.oxide.computer/
Start from RFD 1 ;)
Just because it's hopelessly on-brand for us to offer up a podcast episode for everything, you may also be interested in our Oxide and Friends episode on RFDs with our colleagues Robert Mustacchi, David Crespo, Ben Leonard, and Augustus Mayo.[0]
[0] https://oxide-and-friends.transistor.fm/episodes/rfds-the-ba...
Bryan you absolute legend. You give the best technical seminars i've ever watched (& countlessly rewatched). Ty for inspiring a generation of engineers. Best of luck with everything at Oxide!
I love the idea. RFD 1 mentions taking inspiration from Golang and Rust proposal processes. The Haskell Foundation also uses the same proposal process, and I love it.
I'm a big proponent of the "writing is thinking" mantra. Unfortunately, in my experience, not all technical leaders value grassroots proposal processes like the Oxide RFDs
the rfd interface looks really nice. I couldn't find the github repo for it, is it proprietary?
Edit: some popup on their page links to https://github.com/oxidecomputer/rfd but it's a 404
Edit2: it's at https://github.com/oxidecomputer/rfd-site
Great news for Oxide. I followed their podcasts for a while but they petered out and I haven't heard much about their products/growth for a while. Sounds like it's still viable.
USIT... what a cryptic website! Is it government-related (like In-Q-Tel) or private? Have no idea...
Gergely Orosz' newsletter contained some background on Oxide in 2024:
Very bullish on this team! Congrats. I’ve been pushing my company to adopt their hardware, and we have!
How do you like using their stuff? I would be interested in reading an experience report
Rooting for this team -- just wish i could afford one of these racks... =)
Same.
I think it brings an interesting point actually.
"Who will buy these", the obvious answer is anyone with a need, but the "standard pizzabox" server is ubiquitous for the same reason that x86 and miniPC's outcompeted mainframes.
((controversial take warning))
Mainframes are objectively better at high uptime and high throughput than rube-golderging a bunch of semi-reliable x86 boxes together, yet, the ubiquity of cheap x86 hardware meant that the lions share of development happened on them.
People could throw a pentium 2 PC in a corner and have it serving web traffic, and when things started growing too much you could add more P2 machines or even grab a Xeon 4socket machine later down the line.
This isn't possible with mainframes, and thus, people largely don't mess with them.
The annoying thing is that this kind of problem has some kind of stickiness effect. If you need a server, and then another, you buy them as you need them and if you're already 20 pizza boxes in; it's a pretty big ask to rip them all out and moving to a different vendor entirely than staging replacements one after another.
So I guess their target audience is the "we don't want to touch cloud" organisations that have a good IT spend that are willing to change vendors?
I don't think I've worked for any of those.
(FD: I'm actually a fan of the Oxide team, and the concept, and I would buy into the ecosystem except I have needs that are at most 3 servers at a time)
A great point regarding mainframes, but isn't it somewhat irrelevant given that Oxide's computer is x86 and mainly (...only?) intended as a VM host? And I assume most people are running things in VMs nowadays, so you can "just" migrate over images to the new system (I know it's not that simple, but it's also not quite as complicated as, I imagine, porting something from a bunch of bare-metal x86 boxen to a mainframe).
Also, I'm given the impression that Oxide prioritizes user experience - their website shows off a clean UI and they presumably have modern, easy-to-use APIs. Mainframes, in contrast, seem like a whole different world - if I convinced my company to move to a mainframe, who would even operate it? I know modern mainframes are closer to "normal" servers than their old reputation, but still, I'd imagine it's pretty esoteric stuff, and IBM is famous for not being the cheapest to work with.
I do find it pretty funny that their business model seems to be reinventing mainframes, but I feel like there are important distinctions too. Hopefully they do well (I'd also love to have access to this stuff, but yeah, same "needs that are at most 3 servers" deal).
Mainframes are the original VM host. Oxide racks seem closer to midrange computers from a RAS (Reliability, Availability, Servicing) perspective, but that's pretty much to be expected to begin with. They also have a lot of scope for improvement and are kind of a natural candidate for eventually intruding on that market.
Not very controversial, tbh. Your observation is essentially that there is momentum on a current platform, which yields availability, pricing, and general convenience benefit. It’s borderline indisputable!
The market is complex. Those who will buy will be those who find that the existing ying doesn’t snap perfectly into their own business’ yang. They’ll be at the margins first (the post references a lab for instance, not a booming tech company), then over time less so.
> So I guess their target audience is the "we don't want to touch cloud" organisations that have a good IT spend that are willing to change vendors?
Companies do modernization/migration projects from time to time; I guess one way to solve the audience issue is to find companies that have such a planned event and try to market a “better” alternative.
While I’m also a fan of Oxide; my primary concern is whether they can actually get companies to ignore the marketing that comes out of cloud services.
The target audience as I understand it is companies that have gone cloud-only or close to it and are big enough where moving workload on-prem makes financial sense.
They can migrate to an Oxide "cloud" without too much difficultly as opposed to procuring, installing, and maintaining the rube goldberg machine you mentioned.
They also attract interest among the "we don't want to touch cloud" organizations where trying out $1M in hardware is a rounding error, but I don't know how much traction they'd end up getting.
I guess if you have big pile of pizza boxes, buying an Oxide could feel like buying Oxide Family Pizza and going from there. Maybe you don't migrate everything at once.
Anyone here using oxide hardware? I remember reading their blog post when they were spinning up and it seems like they have actual products now.
They have for a while already, see https://oxide.computer/blog/the-cloud-computer (https://news.ycombinator.com/item?id=38023891).
Kudos & godspeed!
(random superficial comment: the ascii art + high fidelity ui combination on the oxide landing page is chef's kiss)
Is there a clear hardware upgrade path? Couldn't see it obviously on the website.
The on-prem cloud is definitely the holy grail for many of us, but the justification to the capex and ongoing commitment to owning the hardware would be that it has have just as easy an upgrade path.
Hey! The current generation rack is compatible with the current generation compute sleds, switch, and power shelves. The next-generation compute sleds will be compatible as well. Possibly the next-generation switch but I would have to defer to my colleagues for that answer.
We'll ultimately ship a next-generation rack to take advantage of changing constraints (e.g., physical form factor, power requirements) and we'll provide a mechanism for upgrading to that generation of rack as well. Likely, in the form of multi-rack connectivity to move workloads or perhaps even financial incentives to purchase the latest and greatest.
I don't work on the hardware side at Oxide so I don't have further details to share. Great question!
>Our thesis was that cloud computing was the future of all computing
just want to give a shout out to my old boss from the mid 80's who had researched it (mgt consultant) and told me at that time that cloud computing was the future of all computing because the economics were inescapable.
RIP.
I am wondering what they have on Roadmap and if Zen 6 will come. Their AMD EPYC™ 7713P is 4+ years old already. Or is Hardware performance not a main focus for Oxide but Software that came with it?
In 2027 - 2030, We will have 256 Core Zen 7 CPU with PCIe 6.0 or 7.0 SSD and Network. If Liquid Cooling ever come to Oxide we are looking at 5 - 10x the compute power of its current hardware.
Somewhere along the line a Single Oxide Rack would offer enough Compute and Storage for 95% of customers. And whenever I think about having Solaris in every rack just put a smile on my face.
> Or is Hardware performance not a main focus for Oxide but Software that came with it?
It's more of that we're a young company, so we can only do so much, so quickly. Plus, because we do so much custom work, that all takes time. As we grow and scale up, we'll be able to move forward more quickly on updates on the hardware front.
at any rate, I'm looking forward to decommissioned Oxide racks making it into my homelab for pennies on the dollar
They've been working on Zen 5 Turin for a while so hopefully soon they'll only be 1 year behind.
What is the initial cost to setup their smallest system on prem?
Not finding a ton of information about it, was curious about it too. But the "smallest system" seems huge, like 16 sleds and a HN comment from 2023 (https://news.ycombinator.com/item?id=38498840) says:
> An oxide rack has a minimum cost of something like 600k not including all the infra you need to run a rack, maintenance, and then needing to upgrade
So maybe not super useful for people who are looking to do some home-lab stuff or even setups for SMBs.
Edit: Another thread: https://news.ycombinator.com/item?id=36552556
> My guess is that a base config would be somewhere between $400K to $500K but could very definitely go up from there for a completely "loaded" config.
Meanwhile in VMWare-land, you might pay that much in subscription fees with no hardware. https://www.reddit.com/r/vmware/comments/1h8brs2/i_literally...
Wow so really for F500 and such
It's much broader than that, but you're right that we make sense for larger organizations, not smaller ones.
If you have to ask.... :)
(other people's) previous guesstimates were 500k - 1M. It does look like you can order it partially provisioned (compute wise I'm guessing) and expand later.
We're working on publishing an updating pricing page to better communicate this to the public. Stay tuned!
Here's what I can knowingly share. We sell full-rack and half-rack SKUs today coming in at 1024/16TiB and 2048/32TiB of CPU cores and RAM respectively. Disk varies depending on the drive size. On top of the cost of the rack you'll pay an annual percentage for support and that's it. No software licensing or other weird fees.
I googled this out of curiousity a little while ago, it seemed to be 600k or so. (possibly dubious source so ymmv) [0]
Congratulations on the achievement.
It is always a pleasure to follow up with Oxide on their podcast, their technological decision to keep the Solaris linage alive, all the places across the infrastructure they have been using Go and Rust as well.
I can tell Bryan wrote this by the vocabulary... Congrats on the traction - haven't had a use case for rack-scale infra yet but hopefully soon!
I always believed in Bryan, but the day i heard the buzzword "cloud repatriation", I know there was a market.
Big congrats to the Oxide team.
It’s funny how we are knee deep in the next “big thing” (AI), and still haven’t made the last “big thing” (cloud computing) generally accessible.
It should be trivial to create my own S3 bucket that’s stores data on my laptop.
I’m really glad folks like Oxide are moving in that direction.
But it seems when there’s enough money in something, openness goes out the window.
Building sovereign systems management with low overhead is an incredibly important goal. For this I salute them for pursuing that path. However their mistake is that it's a hardware problem. It's actually almost entirely a software problem, modulo convincing some hardware vendors to make their ILOM/IPMI implementations not be buggy or suck.
Disclosure, I work in the software automation space.
Our thesis is that it's both a hardware and software problem! Which, as steve said, is why we're doing both. We can look at designs across the whole stack, from chips chosen, FPGA implementations, hubris management and control plane software to accomplish this goal. By doing these together, we can accomplish fundamentally different designs and tighter integration than any one of those multi-company silos can even if they tried to work together.
We are doing both hardware and software. We don’t use vendor IPMI stuff, and have instead implemented our own stack for it. You’re absolutely right!
I appreciate the craftsmanship of Oxide — it’s like a finely handcrafted Swiss watch.
I just hope they can survive in a world dominated by relatively inexpensive, general-purpose Apple Watches.
I remember when OpenStack was a big thing, what happened to them? They were everywhere- free swag at events.
So what is Oxide and why would I use that over OpenStack?
OpenStack is software. Oxide is selling hardware + software, made together. We believe that's a better approach.
Always happy to see Oxide Computer succeeding. As others have mention their podcast is truly great and their productive is quite innovative in the space.
With this news I hope they plan on expanding hiring for frontend devs soon, would love to work with such leadership once in my life.
I applied here. Basically anti-llm long form one-sided interview writeup.
Took them 3 months before a "we're not interested" email was sent. No reasons, either.
I probably should have just used an LLM to generate good sounding garbage. Probably the same chance to get even a stage 1 interview.
I applied in early 2021. Getting rejected from any company carries a sting, but I was grateful to have gone through the process.
I didn’t realize at the time, but Oxide’s application process was the best form of interview prep I’ve done. The process forced me to thoroughly document my values and career accomplishments. In later non-Oxide interviews, I effectively recited what I had written my materials. In that way, it has felt less one-sided than every other company application process I’ve gone through. I was able to take away an artifact from the experience, versus being filtered out via a coding challenge. It’s also been rewarding to reflect on my submission from years ago to see how my mindset and skills have evolved.
If you have any interest in working in the pediatric telemedicine space, I encourage you to email me your application. We accept Oxide materials. I’m happy to provide feedback as a hiring manager. My email and our company website are in my bio.
Loved reading your experience here. Thank you for posting it. I've written about the value of an artifact in the past when people pushed back against the Oxide materials saying they are a lot of work for no guarantee. When I first applied to Oxide I was also rejected and the materials process taught me a ton about myself and changed the way I viewed job searching and my work. I shifted course and increased my skills and next time I applied I got an offer. There's power in the critical thinking and writing the materials force out of us.
> they are a lot of work for no guarantee.
Well, its the assymetry of wanting a 10 year long documented CV with various orthogonal points in your career, versus actually having a 30 minute call.
Unlike an actual interview, which is equal time investment, this 20 page paper gets the commentary and result of "no". "No" what? You can ask an interviewer about concerns, and discussion points. This email from no-mail@ is just nothing.
And its not the sting of rejection. I've been turned down, and I too have turned down. But its the mechanistic, dispassionate, legalistic response after months of a "No". And not even a 'What we're looking for is.... '
It's tough on both sides. I understand the disdain of receiving a "no" after putting in hours or days of effort on the materials. Candidates are welcome to ask for feedback on their application but must understand that Oxide is limited in what we can say due to legalities. Hiring is a tricky balance on the legal front. I also understand that it's impractical to give every candidate a synchronous screening call just to confirm whether they should continue applying. Not only for time's sake but also bias. Is it enough for 1 person to do a screening call and decide someone's fate?
No hiring process will ever be perfect but at least, as the experience mentioned previously touched on, the candidate is left with an artifact that they can then use in future applications. Candidates walk away having learned something about themselves. Open roles are also limited, applying for such roles is also optional, and it's up to each candidate to decide how much time and effort they wish to put into the materials.
We're humans here at Oxide too, and we're doing the best we can to ensure the hiring process is fair and humane as well.
That's very interesting because in their episode on hiring practices they said that they hoped the materials would be a valuable exercise for anyone deeply engaged in a job search. Hearing that same feedback from the other side of the process really closes that feedback loop!
We do take a long time due to needing to read all of those applications, but three months is longer than we'd like, so I apologize about that.
We don't do "stage 1 interviews", we only do interviews as the very final step in the process, when we've narrowed things down to a handful of people. That initial packet is like 85% of the process.
> Took them 3 months before a "we're not interested" email was sent. No reasons, either.
Yeh this is really really tough. They have an excellent RFD that explains the hiring process and it contains a section on this here: https://rfd.shared.oxide.computer/rfd/0003#_rejection_of_non...
I ported their Hubris kernel to RISC-V and ARMv6 before applying for a role to work on it, so thought I had reasonably strong materials, but got rejected.
I fully understand and empathise with their reasons for not providing feedback, but it does mean you're left not knowing whether you were a hair's breadth away from getting in but were behind a better candidate and it would be worth trying again in the future, or rejected for other reasons that no improvement in materials will make up for. As a fully signed-up member of imposter syndrome club I'll obviously lean towards the latter ;)
Love what they are putting out in the world! Congrats on the round, and being able to proliferate the work!
Isn't it sad we're not able to invest into most new tech companies these days, with private equity taking the lead. I don't blame the companies after seeing what going public entails, but still unfortunate.
some aspect of that picture are nanny-state sorts of laws like Accreditation which have a bias that says if you're "poor" then you're not intelligent enough or qualified enough to make your own best choices of investment.
I was happy to learn about Sweater Ventures https://www.sweaterventures.com/our-story (and their kind) which are opening up access to investments, and also helping ensure the entry price is quite low (for example I invest $50 a month with them) .
It is my hope that in the future you will be able to order a micro fraction of any company as easily as you could a starbucks.
And IMO part of that equation might be to have the state start to work against contracts which restrict your rights on your own property (essentially contracts restrict when you can sell your shares, usually not until IPO or a company organized liquidity event)
There are actually a lot of ways to invest in smaller startups now, but the catch is that the best startups don’t want your money.
Even if all of the laws lined up just right, it’s unlikely that a company like Oxide would be interested in collecting a lot of little investors and then maintaining all of the obligations that go along with serving those investors.
I know that this isn't their mission so I'm not really complaining, but man, if they ever start to offer smaller homelab-scale stuff, they can have all my money.
One POV is their model doesn’t work for small purchases period. It’s not so much a hardware or software vendor so much as an IT agency. Which can be good venture businesses! Of course, everyone looking at this problem thinks, to make a small cost offering: that’s what a cloud is. Oxide’s problem statement says something like AWS doesn’t “make sense.” It makes way more sense than Oxide. The problem with AWS isn’t that it is bad, it’s that Amazon is greedy.
After scrolling the site for 5 minutes, I still don't understand what they do. Do they offer on-prem servers? Then how does it bring cloud scalability? Is their selling point modular computer components so that you can order things online to achieve cloud scalability? Whatever it is, I expect it to have a poor user experience if I can't find a simple 'about' page.
> Do they offer on-prem servers?
Yes.
> Then how does it bring cloud scalability?
You buy an entire (or half) rack at a time, and then can treat the entire thing as one elastic pool of resources, you do not need to manage the individual sleds within the server yourself. Your interface to the rack is similar to a VPS provider, it's "give me a new vm with these specs."
Happy to elaborate if you have more questions.
What do they need so much capital for?
I think the downvoting on you is a little harsh. TFA does allude to it, but doesn't explicitly answer your question. I presume the implicit answer is here:
> With growing customer enthusiasm, we were increasingly getting questions about what it would look like to buy a large number of Oxide racks. Could we manufacture them? Could we support them? Could we make them easy to operate together?
i.e. they need the capital in order to be able to satisfy large orders on sane timeframes - but that's very expensive when you're a hardware business.
Thanks. It was a genuine question but I guess I can see how it might be taken otherwise.
They are a hardware company. Hardware costs a lot of money to innovate and build on.
Love the blog post with a one-off art direction!
Also love the Oxide and Friends podcast! What strikes me most lately about Oxide since falling into their hefty episode backlog, is their book club culture. I really appreciate the ability to get a fly on the wall experience of it, I learn a lot!
Huge congrats to everyone at oxide! I could not be happier to see such a cool company trying to create a real alternative to renting from the hyperscalers.
Is this a government run VC firm? Next gen In-Q-Tel? Makes sense for highly classified workloads on-prem and there’s a lot of demand for that.
Could be a need for unclassified workloads also… but curious if this is a defense and Intel community venture fund backing this next round with my tax dollars.
Likely government-adjacent and with high portfolio use case match. These orgs are less and less keen on shifting sensitive workloads to public clouds - even "gov clouds". Let's not forget that the best access control is to have no access, and that obscurity is actually a form of security especially in a world where effort put on vulnerability searching is proportionate to popularity. The odds of someone discovering and exploiting your obscurely coded and privately hosted oxide stack are orders of magnitude smaller than someone compromising your public cloud.
USIT > U.S. Innovative Technology (“USIT”) is an investment firm that backs growth-stage commercial companies with critical technologies relevant to the national interest, including, artificial intelligence, future of compute, new industry, space & communications, bio & healthcare, and defense tech. USIT was founded by American technologist and visionary investor Thomas Tull.
joined by: Riot Ventures, Eclipse, Jane Street
I predict that if they take this same philosophy and use this money to add some new pieces on top where those new pieces are 'open' in the same way as their current hardware stack but which allows them to also run 'gpu bound' workloads well -- then I suspect they will make a ton of money.
This is the first time I learned about their differentiating factors. Pretty cool.
This is great. I hope they stay committed to the open source side of things, but all evidence seems to suggest they're serious about it. Generally, it's great to see a good idea getting executed well and arguably improving the state of affairs in computing, and making it, as it were.
The flat salary structure at a generous level (from my perspective, anyway) is a breath of fresh air. Everyone getting caught up on the equity is a bit hard to understand, given the clarity of the message from the company.
I will be applying for one of the open positions. Kudos to this company for their approach to business, and congrats on the success.
Always refreshing to see people who actually believe in software freedoms (and not just doing open source cosplay like many big corporations) forge a pathway to big success.
There are many things that suggest free software and the movement for software freedoms might be on its way to a historical footnote. This is absolutely not one of them.
Hey Bryan, one day when you’re very successful market-wise (you and your team have already obviously been massively successful from an engineering standpoint) and aren’t in crash-priority-override mode to get cash flow, please consider a project to build SME stuff that reaps the security and integration benefits of your big enterprise stuff that is affordable for end users like entrepreneurs and home hobbyists, like Ubiqiti does. I’d love a lil’ $5-10k homelab unit, and I bet a number of smaller universities and organizations would go for stuff in the low 5 figure 2-3kW range. Obviously your bread and butter comes from companies that size their orders by number of racks, but if you never go downmarket then thousands of us hackers that love what you’re doing will never get to touch Oxide stuff except at a job in a megacorp.
It would be a good onramp into their ecosystem. Compare this to the deep educational discounts and the school-targeted platforms from Microsoft, Google, Apple, and such.
The term "open source" was created by corporations in opposition to "free software", as a watered-down version of the latter. Open source itself is already free software cosplay.
I see no mention of GPUs in their platform?
My personal predictions:
#1: Don't expect to see it until after AMD or someone else makes a product that actually manages to compete against Nvidia. Nvidia is pretty hostile to not running their low-level software stack, and Oxide is all about the legacy of Solaris.
#2: AMD MI400 or relative will be an extra chipset on future server motherboards (not a separate PCI card). Simultaneously, the boundary between "CPU vector processing" and "GPU used for transformers" will blur, and the chipsets will slowly merge into chiplets in one package.
#3: AMD MI400 and such AI accelerators will be primarily sold as full racks with its own custom "networking" (UALink switch), and the actual host CPU on those devices will be lower specs and mostly relegated to setup and metrics of the AI work, much like storage and networking appliances are built, AI workload will not even pass through the host CPU. I'm not sure Oxide can compete in that world. The "business logic CPUs" will reside in a different rack.
> AMD MI400 or relative will be an extra chipset on future server motherboards (not a separate PCI card). Simultaneously, the boundary between "CPU vector processing" and "GPU used for transformers" will blur, and the chipsets will slowly merge into chiplets in one package.
Isn't this just an iGPU? They generally have much lower performance than GPU's sitting on a dedicated card.
It's built to be deployed in units of racks, so that "i" is a little funny. It's not a little wart on the side of a general-purpose CPU.
432 GB memory at 19.6 TB/s are the claimed specs. For one package. Now fill a rack with 72 of those.
(If only AMD could execute on the software side...)
Earlier answer from Oxide employee: https://news.ycombinator.com/item?id=39182660
If I was interested in getting started with Oxide, could I even do so? It seems like this is only tailored to large enterprise sales. The only other option is just using them as your AWS replacement.
> If I was interested in getting started with Oxide, could I even do so?
https://oxide.computer/remote-access
> It seems like this is only tailored to large enterprise sales.
Since we sell half or full racks at a time, it's true that we're geared towards larger companies, as small shops just don't need that much computer.
> The only other option is just using them as your AWS replacement.
That's the use case, yep!
Congrats to the Oxide folks, like a lot of others I'm rooting for you.
Big bump up in open positions now. Unfortunately none I'm particularly attuned for.
If only they bring back their "On the metal" podcast. That podcast scratched an itch I didnt knew I had .
Check out Oxide and Friends[0]! We've been doing it for several years now, and it's a much more flexible format that allows the team to be heard in its own voice -- and allows us to weigh in on whatever's on our collective mind.
is this just a purchasable version of Borg? kubernetes on steroids?
Congratulations everyone at Oxide!
I would really love to run their control plane in my homelab.
I'm not aware of folks outside the company running the whole control plane, but people have definitely gotten parts of the system running at home:
https://artemis.sh/2022/03/14/propolis-oxide-at-home-pt1.htm...
(The author of this blog post now works for Oxide!)
How much of the software stack is even possible to run without the specialized hardware? I'd also really like to try that control plane..
I love following their adventures. Glad to see that it continues.
> We did our own host hypervisor, assuring an integrated and seamless user experience — and eliminating the need for a third-party hypervisor and its concomitant rapacious software licensing.
In exchange for your own hardware purchase cost, in practical terms also a license.
> We did our own integrated storage service, allowing the rack-scale system to have reliable, available, durable, elastic instance storage without necessitating a dependency on a third party.
In exchange for an unbreakable dependency on the first-party solution.
I'm being overly aggressive because I do lurve your product and the Sun/Apple way of vertical integration is especially valuable for security ... things break at the interfaces and since you have absolute control over that, you can be actually good. Then there's the improved UX that comes with an integrated product. The root of trust work you've done is especially noteworthy, in a sea of also especially noteworthy efforts across the entire vertical product.
But I'm leery that with the absolute lock-in, and VC pressure, you might succumb to squeezing your customers a la AVGO.
They're a cool team and I like the idea so I hope it works. In our case, for the few million, we could get a hell of a lot more hardware (Epyc 9654 based machines to start with - much better operating cost / compute) so the magic must be in the software.
It is true that because we're a young company, we currently have the initial SKU that we started selling two years ago. Like every hardware company, we'll be releasing new products with new hardware every so often. Totally hear you if the current product doesn't fit your needs though, that's just going to be the case when we're so young, but as we grow and scale, we'll be able to release more variations that could make sense for more people.
There is so much more than just slapping servers in racks when you reach the hundreds of thousands or millions in server hardware. Good control plane software makes a night and day difference.
Admittedly this would have been an exotic for the Bay Area but mundane for elsewhere use-case since we just needed really fast disk and fast compute because we just ran model tuning and backtesting on the machines. 90% utilization, 100G network, 4xNVMe. We only had an 8-member team so much of the management/IT-layer replacement stuff that Oxide enables wouldn't have been as high leverage across the tens of thousands of cores. We almost certainly left a lot of stuff on the table, though.
Also, is your username something interesting? It feels familiar but a quick `echo -n '' | md5sum` didn't yield anything.
It's a random sequence.
I suppose I must just have seen you post here before and that's why it's familiar. Funny, felt so familiar.
Congrats!
In the end, the mainframe wins
They got $100M from USIT, which seems to be owned by Thomas Tull
Amazing, considering they don't seem to have GPU offerings.
Lots of praise not much skepticism, so what is the exit here?
$100M is a lot of money investors want to see returned of the total amount raised of almost $200M?
Remember Oxide is VC backed so there must be some strings attached here, is it an IPO or an acquisition for an exit or just staying private?
From the article:
"Our thesis is [...] and that a large, durable, public company could be built by whomever pulled it off."
So the goal is an IPO.
I'd assume eventual IPO. And/or specific lucrative contracts within US Govt/Military/Tech.
AFAIK staying private is nearly impossible after taking VC funding…?
Also, what they do is very capital-intensive, so I’m not surprised that they’re raising more money.
I don't get it. They're a hardware integrator with a secret sauce management layer on top. I like anything on-prem but this seems a bit hyped. Slick website and appears to have a very good team though.
The server and switch hardware is designed in-house (from the PCBs on up), though we do source DRAM / SSDs / CPUs / ASICs from the usual vendors.
The "secret sauce management layer" is available at https://github.com/oxidecomputer/omicron, released under the MPLv2 license.
(I work at Oxide)
These days there are very few companies innovating in this space. Oxide is the only one I can think of. (No, HPE and Dell doesn’t count, not in my book at least.)
There used to be lots of them, but they all had a very rough time after the 90s when cheap x86 boxes started to become ubiquitous.
I have no idea if Oxide will succeed or not, but I sure hope so. If it goes well they might become the Sun of the 20s.
Do hardware integrators usually build their own BMC, their own power supply, their own backplane, their own firmware, their own motherboards, their own switches, their own SDN, their own hypervisor, their own OS, their own rack design, their own blade / sled design, their own management API, and build it all in coordination with each other to produce a comprehensively new computing platform?
In my mind they’re much more like an SGI, Sun, IBM, DEC, or Apple type of play than merely an integrator.
They created a lot of hardware and low level work. (The bullet point on the blog post)
what a weird take.
they produce a private cloud in the form of racks you can buy then just plug in to power and ethernet and run. also, they did a bunch of work so the OOB management isn't fucking terrible, and it uses way less power per FLOP because they bothered to, for example, "make the fans work properly".
Dell will sell you a thousand servers you can then buy racks for, then rack yourself, then buy switches from Aruba, then plug all the switches in to all the computers, then pay VMWare for an vm-cluster-OS, then you can install that, then when something goes wrong you get to call up Dell, Aruba and VMWare and have them all tell you it's someone else's fault.
you...don't get the difference between these two situations?
great people and vision, but the hardware market went apoplectic for GPUs at scale just as they were pushing a better way to manage VMs
in the midst of everyone making a land grab for GB200s, does anyone have time to evaluate their alternative OS?
I don't understand what this is after reading the homepage.
You buy a rack of servers from us. You wheel it into a data center, plug in power and internet, turn it on. You now have a cloud in a box that you can spin up virtual machines in.
That's the high level, happy to elaborate if you have more questions.
A few years ago The Register described it as a rack-sized blade chassis with hardware, firmware, and software designed in-house together.
It’s a 7-foot tall rack with compute sleds, storage sleds, and network sleds. It has its own BMC design, its own power design, its own physical connectivity design, its own virtual networking design, its own hypervisor (although it’s based on Bhyve), its own OS (although it’s based on Illumos), and its own API to spin things up and configure them.
As I think of it, it’s basically a private cloud in a box and The Register wasn’t far off.
Investors are likely betting there is an AI play buried deeply in here somewhere.
The founders are uber geeks but have never done anything successful on their own. I predict this company will be another zombie.
Oxide is a fully closed ecosystem. The product seems cool but it sucks from an open computing perspective.
The “iPhone” of the data center.
What would you like to see open that isn't? The major thing I can think of is not opening up like, our CAD files for the hardware. But that's pretty far away from your characterization.
It's open source (that effectively requires the rack as a dongle) but not flexible or modular in any way. There's not even a single PCIe slot AFAIK.
I see. I wouldn't describe that as "fully closed" but I appreciate the perspective!
God why do startup sites suck so much? Why do I need ChatGPT to cut through the marketing speak to understand what they are actually selling? I literally spent 5 minutes in the site trying to understand before giving up and asking GPT...
Cool 1999 aesthetics though
Maybe I'm old-school, but I clicked on the logo in the top left, then read the first sentence which reads "On-demand elastic resources", which gives me some idea about what it is, and then later it says "A rack-scale system, built true to cloud architecture, that you can own and operate in your data center." which makes me 100% understand what the product is.
Do new internet users not know that the landing page usually contains information about the product they're talking about in their blog posts? 99% of the cases you can find what you're looking for on the landing page, and it took me a whole of 30 seconds to get here, writing this comment took longer time.
> On-demand elastic resources
Which is kind of wrong, because there is nothing elastic nor "on demand" about metal you buy.
The hardware isn't elastic, obviously. But if a IT department sets up a Oxide rack, then the software development department can get the same sort of "on-demand 'elastic' resources" provisioned in that rack. I think that's what they're getting at. But yeah, obviously hardware itself can't be on-demand.
Elastic as in AWS Elastic Compute Cloud (ec2). Flexible virtual machines provisioned with a web API not rubbery stretchy servers
By that metric even VMware's vSphere with its abominable excuses for APIs also count as elastic.
If you have to manage the hardware yourself, have to plan and pay for upfront for the maximum capacity you would need, and there are fixed limits you can hit and have to plan around yourself, it's not elastic.
They make giant rack scale computers for data centers, i think that's the best way to explain it quickly to someone with my background who's never been in a data center but has an imaginary idea of what they look like.
And then as a follow up you have to explain that
- they sell an entire rack full of servers
- they're the only ones that do this. Normally you have to buy all the pieces and put it together yourself, or pay someone to put it together for you but all the parts are kind of designed on their own by different companies so it's kind of a mess. Oxide makes one big rack with like 16-32 servers and it's all designed by them and just works, so you just plug it in and you have servers you can put a bunch of vms on. Oh and they're huge, each one literally costs like a million dollars
They explain what they do pretty much immediately on the home page, I'm not too sure where the confusion comes from.
Reading the first 20 lines of text that they have broken down in to:
Company Name Single line Headline
Problem Our Solution
Gives you the entire company premise and product description.
If you scroll past the marketing blurb, I think the rest of the main page explains in pretty clear and simple terms what their product is. What did you find confusing about it?
Frontpage: "A rack-scale system, built true to cloud architecture, that you can own and operate in your data center."
Seems pretty clear to me.
Isn't Oxide kind of like Oracle now building polished vertically integrated monster machines ? A bit humorous from that perspective given Cantrills dislike for Larry Nevertheless cool company and product.
> Isn't Oxide kind of like Oracle now building polished vertically integrated monster machines ?
Oxide is kind of like Sun Microsystems, building polished vertically integrated monster machines, e.g. Sun E10k.
Isn’t the main issue with Oracle the aggressive sales and exploitation of captive customers?
They don't call oracle "a law firm with a software division" for no reason.
I'd assume oracles main issue is that personal-computers-on-steroids as servers mostly won, and we don't all need AS/400 or mainframe architectures. 99% of problems can be solved with practically commodity hardware and software.
Competing with someone you don't like who has a monopoly on something makes a lot of sense.
To me they are much more similar to Sun than Oracle (before the latter aquired the former).
The company leadership of oxide concerns me. My biggest mistakes in my (not successful) startup were when I got distracted and chased politics or some corporate goal instead of solving my customers real problems. Giving all employees the same pay or advertising social media links to small, political sites makes me worried they've taken the 100 million and are about to crowd the market with a product that's not focused on solving customer problems
What observations have you made that they are not or would not be focused on solving customer problems?
That they're instead focused on standardizing employee salary, or promoting their company through niche political websites.
These are decisions that the CEO is making, which means they're distracted. This is fine when you've received nearly 200 million in total funding, but won't work once you're expected to add the same level of value while being profitable.
> promoting their company through niche political websites.
Just to be clear, you mean that we have a BlueSky account, in addition to X? I struggle to think of what you mean.
Generally speaking, advertising on a wide array of platforms is considered a good thing, as you reach different audiences.
Only your bluesky is listed on your website, which is what people see when they go to your site.
This feels the same as past startups where I thought I hit a pmf because of how excited "potential" customers got when I mentioned politics. We built out "political" features, came back, and none of them converted but the excitement grew stronger. Politics is a trap because it adds no value, drives away potential real customer bases, but you will get enormous positive reinforcement because people are excited you're pushing their brand of politics. In my mind, there's a risk you start creating teams to manage "corporate culture" and brand outreach, and if politics sneaks into management they'll actively steer the company in the wrong direction.
Also I'm sorry, if you're actually an executive at oxide I don't mean to be harsh, I thought I was speaking into the void. I've witnessed this destroy companies I've worked on and been involved with, and doing something like only listing a bluesky account is exactly the kind of thing that's a distraction which costs potential real customers.
I am not an executive, I just work here.
I agree that it's difficult to balance features customers say they want vs the ones they'll actually convert over in an early stage company.
I just don't think "has a BlueSky" means that that is what happens.
A consequence of us all making the same amount of money is that we don't waste time on performance reviews!
Huh, what I read and hear leads me to believe that they will succeed because of their leadership.
Does it follow that because you lost focus that someone else will?
In theory yes, but in practice only so many companies receive nearly 200 million in funding, and they'll be able to buy out competitors and consolidate the market for years to come. Oxide will likely be with us for a while, and in 5-10 years I wouldn't be surprised if people start to hate them while their exec team hangs out with Tom Brady