Hardware is Expensive, Programmers are Cheap
lastinfirstout.blogspot.comIsn't the real story of this "incompetence is expensive"? Seriously, that application design is worse than the worst implementation we've ever had in a failed outsourcing project. (I make Big Freaking Web Apps for Japanese universities at the day job.)
Up until recently universities pretty much had to overspend on hardware, though. (Virtualization/expandable-on-demand cloud computing may eventually get around to changing it, but the customers aren't ready for it and the programming costs to take advantage of it dwarf the benefits for our clients.)
Most of the important systems that need to scale have very, very bad usage patterns from a hardware buyer perspective: for example, take course registration. (Edit to clarify: They are probably not talking about course registration, because there is no way in heck that peak is only 50% more than the steady state.)
At a university with 10,000 students, about 360 days out of the year you can run the course registration on a laptop while it is being used to play World of Warcraft. Then there is course registration season, at which point your peak concurrency goes from 1 user per hour to generally a multiple of your student population all signed in at once. (Because, no matter what you do, they will open multiple windows/connections/etc because "the site is so slow, come on duuuuude, why the heck is this POS always so slow?")
All the accesses are dynamic. Most of them have writes attached. You have to get caching right because if you overbook a class and tell 15 students that they're confirmed a seat in a room which sits 12 because your cache got stale for three minutes, your customer gets yelled at, and they will turn around and yell at you. The end users are also typically incompetent at using the system (typically 1/4 of them have never used it before) and they will perform an impromptu fuzz test on it.
(Oh the stories I can't tell, sadly.)
I used to work at a company providing enterprise systems for higher education. Fall registration was panic season; we were in fire-alarm mode for most of August and September.
Those were good times. :)
eLearning (On line Courseware) for a population of a quarter million students, about half of which use the system daily.
Registration is as large of a system, but with higher peak loads at semester start, as you've indicated.
Sometimes Hardware is Expensive, and so is time though.
I think often times in a startup, especially one thats bootstrapped, its easy to get caught up in trying to optimize everything too early on to scale for a million users before you have your first thousand. Sometimes you have to optimize for your development time first and product development goals, and leave the optimization till you get a bit of runway.
> Sometimes you have to optimize for your development time first and product
> development goals, and leave the optimization till you get a bit of runway.
That's a typical way to postpone and fall into a trap. Not all optimizations take huge amounts of time. And certainly there are some architectural decisions you can take early on that can save you a lot of wasted time later. The most common is to prevent bottlenecks.
For example, it's not necessary to make a clustered DB from launch day but coding and managing the UI and middle-ware to be ready for a switch to clustered DB usually takes very little impact. If you don't do this and the site becomes successful, re-engineering your whole site to support a new DB is usually close to impossible or extremely expensive. The site ends up in the DB hardware feedback trap just like in the article.
Edit: format fix.
Architectural decisions also limit what you can do later. Generally, the more focussed/optimized a system the less flexible. To the extent you KNOW exactly what the system should do, it should be optimized from the beginning; but, as PG points out repeatedly, most startups change direction at least once after launch.
"Had the app been as efficient three years ago as it is today, I'm estimating that about half of what was spent on hardware and related licensing and support costs would not have been necessary."
So first off the software costs where far higher than the HW costs. Second, possible savings of 750k would have come at the cost of features or development time both of which have costs. If anything this just points out how expensive using Microsoft solutions can be when you need to scale. But, even still I expect over 3 years the 750k "lost" is an insignificant cost vs the cost of development and roll out.
PS: Reading between the lines the HW costs where probably well under 150k. "up to 30% of 32 database cores all by themselves."
At the time, IBM, Unisys and Bull were about the only vendors with SQL 2000 capable servers with > 8 CPU's. (And Itanium, of course.)
The customer purchased IBM x460's, 16 CPU's, 64GB RAM = $225k each in 2005, right after they came out. Three required (active, passive in a cluster at the production site and one for the DR hot site). Upgrade them to dual core x3950's six months later = another 200K (or so, the upgrade was to 16 dual cores, 128BG RAM for each of the three database servers ). Plus all the odd's and ends, like 60 amp 3 phase power, 70,000 btu's of cooling, etc.
The software/licensing/support costs were because with > 8 sockets, you needed Microsoft Data Center Edition, and at that time, it came with an expensive support contract ($70k/yr).
Had the vendor been about 6 or 12 months ahead on their optimization, the 16 CPU x460's could have been 8 dual cores instead, and the 32 core upgrade probably wouldn't have been needed - or if it was, it could have been delayed until quad cores were available, the expensive OS support would not have been needed, half as many Microsoft SQL Enterprise CPU licenses would have been needed, and the 24x7 hardware maint contracts would have been half as much, etc.
It’s stunning how much people can overpay on just about anything.
However, even if every cent of that was "needed" we are still only talking about ((3 *225 + 200) /2 ) ~= 440k in HW which is ~1/3 of the stated number and probably a small percentage of the budget over those 3 years + dev time.
PS: I know they would have saved a lot of money by using 8 CPU machines, but that's part of my point. Scaling the DB system has little to do with the app code and it shows just another way the project was miss managed.
In other words, figure it out for yourself in your particular situation. Do the back of the envelope calcs. The reason why articles like this and the opposing view at Coding Horror are posted is because authors want to draw attention to management with preconceived notions.
Looking at things along the hardware/programmers are expensive/cheap axis (take your pick) is short sighted and highly subjective. Setting aside political considerations, the real thing decision makers should focus on are the organization's underlying time and efficiency constraints and unfortunately these things are most often overlooked, misunderstood or inaccurately represented.
This sounds like an edge case to me. Now that it's no longer 1998, most of us don't spec out $500,000 boxes for our stuff anymore. A $5,000 box will get you a long way for just about anything you need to do.
I suspect that the author is looking at a problem that would require tons of hardware regardless of how well it was optimized. Evidence of this can be found in the fact that even after tons of optimization, his $1.5M setup is still running at 20% load steady state.
Our single <$5,000 box handles about 4M pageviews per day without moving the cpu above 5% steady state. That's the sort of baseline I'm used to from the Microsoft stack, so it causes me to question whether the author is really looking at a mainstream case.
> Our single <$5,000 box handles about 4M pageviews per day without moving the cpu above 5% steady state. That's the sort of baseline I'm used to from the Microsoft stack
You realize how meaningless a statement like this is, right? You just can't go around talking about "pageviews" as if they were some uniform measure of workload.
Yeah, I was about to mention that. 4M a day is about 50/sec; if it's nothing but static pages you could serve that on a Pentium 1 without breaking a sweat. I've been giving away P4s recently, so their value is effectively zero - they can probably do it at under 5% too.
The problem is obviously when you're not serving static pages.
This is not a problem of hardware vs software. It's a problem of vendor's money vs your own. Of course he doesn't care. Actually, the more you spend on hardware the cheaper the software seems.
Of course he doesn't care.
My day job sells integrated solutions to Japanese universities. We would care very intensely how much the hardware costs in a circumstance like this. (Which we wouldn't be in, because we try not to deliver software which is an abomination against all that is good and holy in terms of database use, but still.)
The math is simple: the university typically has a budget of X million yen to Get This Done. It doesn't matter what the line items are on our invoice -- we can't charge them more than X million yen.
Given that constraint, what do you think we want to charge them? Software license fees for our solutions, where our margin is anywhere from... crikey, I can't tell you the numbers, but "high to higher". Software license fees for third party providers like a certain Enterprise Database, where the margin to the reseller (us) is from low to medium? Or the margin resellers get on hardware, which compared to our software is small enough we could use it to remove food from between our teeth?
I mean, once the contract is signed. I know how sucky it is to go into such a relationship with a vendor. Custom apps are always risky... and hardware costs are one of the risks.
In your situation, actually in most situations when the client can compare the costs for both software and hardware before, it's logical to optimize the software.
there is a fundamental difference between software for sale and software as a service when it comes to this debate. the economic efficiency math is much different for licensed software because more than one company is using it.
"the economic efficiency math is much different for licensed software because more than one company is using it"
So is the risk.
Nothing drives customers crazier than a slow system because someone else is having a busy day.
I spent my first 10 years decoupling applications from each other because independence was more important than economies of scale. Now we're swinging back the other way. Hopefully, we will have learned something this time.
Yeah, if you can make software run 100 times faster with lots of profiling and hand optimisations, that $300 laptop is basically a $30,000 laptop.
Of course you need to consider what those optimisations will do to maintenance and future development costs.
I think the problem is that the costs add up slowly. Your site starts getting slow? You just throw another dedicated server at it, increasing your cost 300-400 bucks a month.
Hm, why not find something to do with the remaining cycles?
SETI @ Home? :)
The customer hardware is IBM xSeries (3950's), with modular, stackable components, much like a stack of switches. It could be broken up into smaller servers and re-used on other projects.
In a production database!?
I was assuming in a modern "cloudy" setup, processing power and storage is all kind of fluent. Or if it isn't yet, might be worthwhile to make it so? They could set it up all with virtual machines and stuff?