A Billion User Load Test on Healthcare.gov
blog.navapbc.comHealthcare.gov is the absolute worst. They had a button "End coverage for 2017" I clicked it around the end of November because I chose a non-marketplace plan for 2017, and the end result of that click was Cigna my current marketplace insurer for 2016 received a transmission from them, and interpreted it incorrectly, which made my 2016 plan inactive and retroactively terminated me back to 08/31 despite having paid all my premiums and processed claims up until end of November.
Their escalation process is useless, 30 days to resolve (if I'm that lucky). Will try not to get injured in the meantime.
Glitches and data sync issues are unreal, this is not an isolated incident. Every year it's been something. 2014 I was on Covered California and had to take them to administrative court to resolve data and tax form issues (canceled then due to a move and it completely wiped out my 2014 enrollment). Then in 2015 on HealthCare.gov they had me enrolled in the same plan twice and the insurer wanted double the premiums until they could resolve it.
I had a similar problem with BCBS --- twice, actually, the latter of which cost me most of a year's deductible and almost left us without insurance. My impression is that the problem with the insurers, not with HC.gov.
This. I've actually worked with some large payers and they are just literally not set up organizationally, technically, and systematically to work with individuals or even very small businesses for that matter (less than 5 employees). Furthermore not only are they not setup to help individuals, they can't even figure out whether it's financially viable for them or not.
A big problem with the individual market is that it isn't financially viable to serve the individual market at reasonable prices, because in the US market, not holding a job with adequate health insurance is a huge adverse selection marker.
Startup founders should be a lot more concerned about this than they seem to be. Health insurance almost screwed me at Matasano in 2005; it would have, had Erin not taken a full time job elsewhere until Matasano had group coverage.
My wife (startup founder) is under my insurance (I'm a full time employee that gets $12k per year towards UHC healthcare). Even if she paid the same from her company (~15 employees), her deductible would be double what it is currently. If she went under her own company health insurance, we would have to pay at least an extra ~$2,000 this year out of pocket for various health related services she incurred.
From a macro lens there is definitely a huge disincentive to start a company in this country.
I think a lot of 20-somethings think that they'll have the same access to health insurance they get now when they're in their 30s and starting a family.
Well won't that be a surprise when they see their premiums with a partner and children. I know I was certainly surprised to find I went from paying $300/month for myself to $1300/month for a family.
I think they're going to be a lot more surprised when they let their insurance lapse for a few months and find out that the ACA amendments provide guaranteed issue only if you've had continuous coverage --- without continuous coverage, you can be uninsurable without first working a full-time job for a year.
Not "expensive to insure". Uninsurable, at any cost.
You can skip coverage for up to 3 months a year ("Short Gap" exemption). If you do this at the end of a year, you can get back in Jan 1st of the next year without penalty. You could, depending on risk level, only be insured 9 months each year to save on costs.
https://marketplace.cms.gov/technical-assistance-resources/e...
That's under the current ACA. The most likely repeal/replace bill replaces the individual mandate with a far stricter continuous coverage rule.
Now we see the real test of whether healthcare.gov has reached the level of consumer tech: Does bug-report-via-top-ranked-HN-comment get anything done?
After doing 6 years of government contracting and also interviewing with Nava I get the impression they're a higher grade of government contractor than what you typically work with. The initial failure of Healthcare.gov was no surprise to me (in fact I was surprised they even had a homepage working based on my government contracting experiences) and having Nava come in and fix things in what was likely a pretty horrific codebase is nothing short of impressive.
Having said that there is still a ton of red tape and issues surrounding not just government contracting in general but also around the fact that different states and insurance providers have different systems that all work differently but somehow need to work together under this single application.
So I wouldn't expect much even though the Nava folks, in my short time dealing with them and comparing them to other contractors, seem fairly solid.
AFAICT it's actually a Cigna bug report yeah? Happy to make sure it gets to the right place if I'm wrong.
(/me doesn't work for nava, does work on healthcare.gov)
There is a persistent error message on my 2016 profile on healthcare.gov after that button was clicked (despite it also showing it's active on that side of things). My understanding is that the bug is the integration point between the two. From past issues and what I've been told on this one, is that the only way things get fixed and made right is to have healthcare.gov retransmit the correct data/status. Cigna can't fix it on their own from their side. Thanks for the offer to help, didn't expect any action, just a post out of frustration. I'll shoot you and email.
I'm almost positive you mean "I'll shoot you an email." but the idea of someone being so frustrated with healthcare.gov they want to shoot the messenger (yet still get the problem resolved) is in-line with what I've heard. (and gave me a hearty chuckle)
Haha, just a typo and not a subconscious slip, I swear! :-)
Got it, it sounds like there is some issue on the healthcare.gov side. I work at Nava so feel free to email me as well. I totally understand your frustration, and though it sounds like we don't work on the components of healthcare.gov that are leaving you in the lurch just reach out and we can try and route your issue to the right people.
Thanks so much, most appreciated. I'll take the first offer of help first and if that doesn't work I'll follow up with you.
Both Nava and healthcare.gov people replied, in under an hour. Looks like a win for the OP (or as much of a win it can get in this fucked-up situation).
Fingers crossed!
Just wanted to add that both were a big help and that my insurance is active again. Thanks to everyone who tried to help. I'll have to pay it forward somehow.
I wonder if this healthcare.gov will exist a year from now. It is nice to see how far it has come in 2 years.
Doubtful Trump is getting rid of it.
Personally I preferred the previous way of signing up for coverage ... call up a private insurer and sign up? The cost was less too as I wasn't paying for insurance for lazy sally or Joe Schmoe! So for me Obamacare/social medicine forced those who do and work to pay and cover for those who don't. See ya Obamacare(love obama though).
Also the $300 plan I paid each month for sucked.. they fought me tooth and nail to cover things my doc prescribed.. I.e. Various scans(cat scan, etc).
> call up a private insurer and sign up?
For many (most?) cases you can still buy individual market directly with the insurer via phone/website. My understanding is you have to go through exchanges for certain subsidy situations.
> The cost was less too
> Also the $300 plan I paid each month for sucked
First off, the old way was for an employer to handle it for you. If you had your own solo insurance before, I'm impressed. I had a hard time finding health coverage as a freelancer years ago.
But more to your point, the cost was less, and sucked, because insurers could offer barebones, high-deductible plans that didn't really cover much except for catastrophic needs. Hence why you encountered high payments every time you actually needed healthcare.
In addition, premiums cost more now because:
- The highest risk and most expensive people were excluded from coverage entirely, either because they couldn't afford it or insurers refused to cover them, which they are no longer allowed to do. As a result, insurers are paying out more to cover medical treatment.
- Providers and drug companies continue to raise prices, while insurance premiums are regulated and locked annually.
- Government subsidies that were initially promised to insurers to help them cover gaps as they figured out the first few years of the ACA market were eliminated prematurely (cf. Marco Rubio).
As a result, annual increases have jumped dramatically as they adjust to increased care, increasing costs, and drops in subsidies.
So premiums are up because (a) you have better coverage (due to regulated minimum levels of coverage), (b) there are more sick people being covered (either because they couldn't afford coverage before or were excluded), and (c) net costs have risen.
> forced those who do and work to pay and cover for those who don't
I think that's an overly simplistic way of interpreting the situation.
You have to go through the marketplace to get a guaranteed-issue plan.
What do you mean by "guaranteed-issue plan"? All insurers must offer at least the same standardized minimum plans (bronze etc.). Anyone who expects a subsidy must go through an exchange.
Insurers don't HAVE to sell direct to consumer (off exchange), but they're incentivized to do so because that population is on average healthier than that coming in through the exchange. Plus it's easier to manage customer records directly than rely on the data that comes from the state or federal exchanges.
Based on my understanding of the purpose of the Individual Mandate is that it's less that the well employed are subsidizing the under/unemployed, and more that the healthy are subsidizing the unhealthy.
With everyone left to their own devices, the elderly/unhealthy are strongly incentivized to have insurance, and the young and healthy might reasonably conclude that their best bet is to not pay for insurance, that results in higher average premiums.
People waiting until they get sick to sign up, essentially. The insurers' previous solution to this was no coverage of "pre-existing conditions". But people hated this generally, for various more or less good reasons. Banning exclusion of preexisting conditions is one of the handful of parts of Obamacare that's wildly popular across the political spectrum, at least among non-politicians (it polls something like 70-80% support). Obamacare's main replacement for that is the individual mandate requiring you to always have insurance, rather than to go without until you get seriously ill and only then purchase it. (There are also a few secondary things to try to discourage buying insurance only right before you're going to use it, like the open enrollment period being only once a year.)
Huh, I had never realized that purpose of the open enrollment period. It always seemed unnecessarily restrictive until now.
Calling insurers and buying directly is potentially cheaper, because those insurers aren't selling marketplace plans --- the marketplace plans have guaranteed-issue requirements, and must conform to minimum standards that off-marketplace plans do not.
The downside is that if you have anything interesting on your medical history --- including many things in your past that aren't chronic conditions, but could be predictive of future health care costs in some circumstances --- you will be turned down without recourse. That's the problem with the pre-Obamacare individual health care market: it didn't work.
All the liberals down voting me oh well... I'm an independent and didn't vote for that maniac or that queen!
This is a simple issue of how healthcare Became more expensive, less accessible (primary care docs hard to find and not readily available to see you.. so we flood the ERs), harder to use(insurance companies throw up the gauntlet and don't want to pay for things prescribed) and a real pain in the butt to sign up for.
All the above happened because of Obamacare and its eye towards helping every man, woman and child access to affordable healthcare. But that mantra goes against capitalism and thus creates a system where care is worse because insurance companies who once covered most things aren't so they have enough money/profit to pay for those men, women and children who are poor and can't pay. That demographic is the winner and all others are the losers in this game. It's a losing game either way and soon for the poor so and again see ya Obamacare ... socialistic ideas working within a capitalistic system is going to fail.
If it was completely socialistic it might work, but heck no the govt already taxes me enough I don't want them taxing me at 60 to 70 percent!
Hopefully it is replaced with a better system! Time will tell.
*typed on crappy iPhone 7 whose auto correctness/AI needs work!!!
I'm paying $400 a month now thru my employee just for myself with 1k deductible... that's nuts.
My $300 Obamacare plan was horrid! Love Obama but see ya Obamacare!
nuts? I'm paying $700/month for a family through an employer plan. The cost of the worse plan on Obamacare is about $1500 (oh, and before obamacare that plan would cost the same $1500 and come with pre-existing conditions crap).
Medicare for all, $120/month/person - problem solved.
From the information provided I am doubtful that this is really a load test of healthcare.gov. It sounds like they are testing the login and account management system in isolation from the overall workflow of entering your financial data, dependents, etc. and shopping for plans, which is where the sausage gets made. It is an interesting post but it seems like this is about the most trivial component of the overall healthcare.gov solution, unless I am missing something.
8K transactions / second is hardly extreme load. It's good they can support it, and I doubt they need more, but many, many sites handle more load than that.
I will say that if the system scales well horizontally the important metric is how many resources you need to achieve that amount of transactions.
If they used 70 4-core m3.xlarge that means that they can attend 114 transactions/second per 4-core server. And that looks slow without more data about what are the requirements for the login process.
But if you look at the peak of 150 requests per second, then probably it will cost more money to improve the performance than to actually just add servers.
At work we have over 1 billion users. But as it scales horizontally the relevant number is how many memory/disk database instances do you have. The same applies to the application layer for the number of transactions. In fact the development servers are slower than the live ones even that they have just tens of thousands of users. For our amount of users/transactions tuning for performance saves lots of money. So it makes sense to improve the code on that measure.
For comparison, when I received the RFP for the Italian and the British (gov uk verify) national authentication systems.
The numbers that were asked for were 12k/auth per second during peak time for the Italians.
The British wasn't clear but it hinted at 0.1% of user authenticating per second, which is about the same range. (The numbers of users itself was not defined, it depends on who would join the program).
SLA: 1500 ms for the account verification. 150 ms for most operations. (Some more time when requesting optional attributes like name/address/phone)
If this interests you, I've published a few numbers I got from preliminary testing (to determine whether we could answer the RFP or not): https://thehftguy.com/2015/10/23/10-millions-users-accounts-...
Agreed, it is good they can support it. However, is 8k TPS really a big deal? The team I work with handles 1M TPS.
To be fair, a "transaction" here is probably a lot more complex than people give credit. Healthcare is a very sophisticated beast... it's a huge many to many problem, and though people keep trying to "standardize" details always prevail.
Most sites are more complicated than they seem from the outside. Complexity is fractal -- the closer you get to something, the more complex it is.
One of my past employers was Salesforce.com. Seems pretty simple at first glance (and the early implementations were very simple), but then you learn more. Ignoring all of the platform-related stuff, just doing a multi-tenant database at scale turns out to be plenty hard.
The fact remains, they're a couple orders of magnitude away from extreme load.
That is a really good point.
it's the government, we're thrilled they can do anything.
I know it's just a throw away joke, but this persistent idea that government is incompetent, no matter what is really damaging.
When it comes to health care systems there is no evidence whatsoever that less government means more efficiency, if anything the US is a huge example to the contrary.
Im not really joking, their ability to do anything efficiently or correctly for a reasonable sum of money is embarrassing. I know it's a joke as well, but it's funny because it's true.
Social Security has vastly lower overhead than any equivalent private program.
Now, you can see crazy waste on both sides EX: Drug Advertising. But, it's hard to look at this objectively.
My experience is the opposite. Private companies seem completely inefficient compared to government, and I've worked as engineers in both.
Really it is only a right-wing profit-driven talking point to say government is inefficient, mostly to protect against their profits that government takes away.
Government is exceedingly efficient.
I've worked in both as well and vehemently disagree.
The numbers support my case. You can send a mail with the USPS for 50 cents anywhere in the country, but it costs $5-$10 to do the same with UPS or FedEx.
Literally 10x-20x costlier.
Government just doesn't have to waste money on things like profit or marketing that private companies need to do.
USPS is a private entity under the government, they fund themselves. They receive no funding from the government.
You can look at the military and at obamacare/heathcare.gov for examples.
You can also look at the USPS as an example, since it IS a government entity that doesn't have to serve profits.
Also, the military and healthcare is more efficient in government, but you probably already knew that already.
Uhh, the US has had extreme government regulations regarding healthcare for as far back as I remember. Every state has it's own insurance administrator and purchasing insurance across state lines isn't allowed. What we have right now is the worse of both systems actually. It's a heavily regulated false free market where the customer has basis for comparing prices and has no direct control over providers. Instead they must choose between a government chosen option or an employer chosen one. It's just terrible all around.
With a single payer system, at least it would be fair to lay all blame on the single payer. It'd still be beuqacratic and political, but at least there'd be direct responsibility.
Removing the regulations, at least the customer would be able to choose a provider themselves and people could sue over pricing disagreements. There would be issues with pre-existing conditions and low income that would still need to be addressed, though.
Either option is better than where we are today, which is, well I'm not sure. There's not really a term to define the mess we've created. It's not socialist, it's not capitalist, it's just a mess of special interests, political concessions, and reactionary legislation. I guess the one thing we can definitely call it is bureaucratic.
It all depends how much politics are involved, or how much public workers can freely do their jobs:
https://www.nasa.gov/ http://www.darpa.mil/
And is Healthcare.gov developed by the government or just bough to an external company? It just an observation, I don't know if Healthcare.gov is a good or bad technology.
Development is contracted out, it usually go along the lines:
Budget $1 Billion
1. Large government contractors: keep $400 million, outsource all work to->
2. Other government contractors: keep $200 million, outsource all work to->
3. Third party companies: keep $100 million, outsource all work to-->
4. ....
5 .....
..
10. Bangladesh code farm does all coding for $10 million.
I believe it was one of the first projects of the digital service. (https://www.usds.gov/)
Not quite. It was designed by a contractor, developed by several others, and once it was released to the public and shown to be dreadful, another contractor was put in charge of fixing it. It's still primarily developed by contract to this day.
The first head of the USDS was involved in coordinating the fixing of Healthcare.gov, though.
That's probably what I was thinking about, thanks.
Especially after spending billions $$ just to create the platform, it should have been able to do that day 1.
You know 1M select(1)/sec is easy, now 1000req/sec with complex process is not easy.
The number of requests per seconds doesn't matter, what matters is the complexity of the logic behind the request.
I like that you are assuming the complexity of 1M TPS without any context what so ever.
Yes 1M req/sec must but pretty simple, the only time I've seen those numbers were definitely not like "login calls", let me guess advertisement?
At that rate you could enroll every household in America in 3.5 hours.
I'm glad heathcare.gov is getting better. I would assume however the most important part of tests for open enrollment are the writes to user management where reads and writes simultaneously are exercised against datastores. Looks like users were manually created not sure if they included updates in the tests. I'm wondering if the tests were just auth based? That would make sense why they could easily scale horizontally since auth is a lot of times just limited to cpu capacity.
A week to fill a DB and 70 m3x-large is way too expensive to find out how far you can scale. Me and my friend built stacktical (https://www.stacktical.com) so that with very minimal load testing it's possible to understand and share how many concurrent users your current and upcoming application infrastructure can handle. Testers are welcome
7754 tps over 1 hour is 28M. They may have 1B users in the db but they only tested activity on at most 28M.
True, but running at 7754 tps over an hour with a database of 28M users is easier than with a database of 1B users.
It is much easier to test with a small # of active users and a large (mostly unused) database, than with a large # of active users. To me the title implies 1B active users ("billion user load test").
They are two different problems.
If you have many active users, you're testing for the write load of user tokens and sessions.
If you have many accounts, you're testing for the read load of the account database.
Technically speaking, user id and sessions should be two different database systems, because they have different requirements and they're conflicting.
For the first one, the problem is to have lots of write and enough performances. Plus having the storage if you keep tokens forever (typical for auditing).
For the second one, the problem is to be able to store all the accounts, plus read performances (which can always be hacked by adding layers of caching).
Funfact: if each account is only email + password (64 + 64 char), 1 billion accounts consume 128 GB, just for storing the ASCII characters. A lot more in practise with metadata, empty space and other fields.
They say that they only had 1 maria DB on a single EBS P-IOPS volume. That's kinda weird that a 1 billion users database could fit in that -and zero errors, really?-. Maybe they just had a single integer field (1,2,3,4) to define the user and the password :D
Next thing to look at is why only 2k/sec users can be added to the DB.
The website is terrible; they keep spamming my e-mail and calling my number with the same alerts over and over again.
I'd like to see this kind of authentication done over OpenAM and OpenDJ. That's the IAM systems done by Sun Microsystem (later Oracle, then Forgerock). That should be rock solid.
IAM: Identity and Access Management
Forgerock: https://www.forgerock.com/
> The current observed peak load for HealthCare.gov’s Open Enrollment in 2016 was on Dec 14
2016 Dec 14? Did I miss something?
Probably Dec 14 2015 for the 2016 enrollment year
What's it cost to run Healthcare.gov on AWS, how many people-hours were spent in designing, configuring and deploying the website as compared to a non-cloud solution?
Why load test for > 3x the number of potential users?
> (of only 2 machines!) with a total of 6000 worker threads.
What about the bugs that arise from many connections from many machines? Those must exist too.
I am sure my wife will be very happy to learn about this test. It will definitely make up for our insurance going from about $12K per year to $26K per year. That, along with keeping our doctor and saving $2,500 per year have been fantastic.
News Flash: Engineers think technology while, in the real world, the problem has nothing whatsoever to do with it.
It's like tuning your Ferrarri for more power while your reality is you commute on the 405. In other words, irrelevant.
But, hey, congrats, glad to see what I am paying for.
The goal of technology is to be so flawless that it dissapears into the background of our lives. When healthcare.gov was crashing the technology was extremely important and it made all the news and became a political controversy. Now that the technology is fine people like you can finger wag engineers and tell them that their work is unimportant.
You missed my point. I am an engineer. When I was younger I, too, wasted lots of time optimizing the irrelevant. Didn't fix a thing.
The ACA/Obamacare doesn't have a healthcare.gov traffic problem, it has a "it's a piece of shit" problem.
Of course, this is not the purview of engineers working on healthcare.gov. The point there is that we all get excited about building monuments to technology while completely forgetting the problem that had to be solved in the first place.
Someone will invariably say something like "20 million people have insurance now that they didn't have before!" or some other fabricated number. Let me address that now.
First. The President of the United States of America promised ME and my family that we could keep our plan, our doctor and, on top of that, we would save $2,500 a year. No need for a link, it's all over YouTube. He made that promise dozens of times.
I don't care of one or a hundred million people got insurance. The POTUS made a promise to families. If someone in business made such a promise and then reality was exactly opposite, in an unbelievably grotesque way, the smallest lawsuit for breach of contract and fraud would have them curl-up into a fetal position and cry.
Oh, yeah, and the rich part is that they stick the IRS on us. You know, the Gestapo-like, most-feared, we-can-ruin-your-fucking-life agency in the US now has hooks into your healthcare. And this is OK, how?
The POTUS lied to all of us. Period. The fact that there are people with insurance now is irrelevant. My family is playing the equivalent of a home mortgage for health insurance. How could anyone even remotely tolerate this from our elected officials.
Now, to address the "insured".
Most of these people have been shoved into Medicaid. The magic pill that solves it all, right?
I am not going to go into a long explanation. ACH/Obamacare represents the largest theft of personal property in the history of this nation. Why? Here, go learn something:
https://www.medicaid.gov/medicaid/eligibility/estate-recover...
Short version: These "insured" have signed over their personal assets to the government. All care they receive --by law-- requires reimbursement and the government is REQUIRED by law to recover costs.
Wonderful, isn't it? So many people now have insurance! So many people who have no clue the government just sunk their hooks into their estate. What a country!
I don't particularly like Trump but I hope he shreds Obamacare to little pieces, it is nothing less than the largest criminal act ever perpetrated on the American people.
A few years ago, pre-ACA, and even without this kind of load testing, my individual high-deductible ("catastrophic") plan with a major provider had a 30%+ yearly premium increase with zero additional coverages whatsoever. They didn't say why at the time, but it was probably just system load testing costs. /s
Maybe you should go complain to your legislators. Engineers don't have the Constitutional authority to enact policy willy nilly.
What did your premiums look like before? In my case I just didn't have insurance because there was no functioning market anywhere that would sell you a family health care plan including childbirth, at any price. Now I get health insurance from my employer which isn't affected by ACA so I don't know what the market looks like.
We were paying about $650 per month and a $4,000 annual deductible.
Obamacare forced our insurer to cancel our plan.
First lie: "If you like your plan you can keep it".
The new plan was $1,450 per month and a $9,000 annual deductible.
Second lie: "A family will save $2,500 per year".
Even at that, we lost many of the doctors.
Third lie: "If you like your doctor, you can keep your doctor".
There's more in the details (co-pays, etc.). The above would be enough to throw any business person in jail for fraud. Yet we somehow tolerate and rationalize our elected officials behaving this way.
I don't particularly like Trump but if anyone is wondering why he got elected, well, start by understanding the true depths of Obamacare and go from there.
Do you think it's possible that your prior policy was one of the garbage plans that got outlawed by ACA? Those are the ones that didn't survive the "you can keep it" claims. Those were plans with uselessly low annual or lifetime limits, too much small print, etc.
Impossible. Not garbage at all. We had the same policy for about 15 years. It was better than what we have now for three times the money.
Fairly certain you're paying for the unemployed person down the street's insurance. (I say this not pejoratively toward's the unemployed)
Insurance companies don't insure the unemployed for free. They're either paying their own premiums, or they're on medicaid/medicare. If they're on medicaid, it was paid for by the state/fed government from taxes... not from insurance premiums paid by someone else.
The cost of Medicaid and Medicare may show up in insurance premiums.
http://kff.org/report-section/uncompensated-care-for-the-uni...
There are other discussions of cost shifting that will make more aggressive claims; I've chosen to link Kaiser Family Foundation because I think they are pretty credible.
the amount potentially associated with uncompensated care cost shifting is only 2.3 percent of private health insurance costs in 2013.
Sure, it doesn't explain the high cost of care.
But the $21.1 billion out of $84.9 billion is a pretty big chunk of the unmet costs.