$5,000 Security Breach
blog.joemoreno.com>'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.
This doesn't make sense at all. Amazon should let us if monthly bill > X send me a priority email and phone call. Why do they hide behind these dark patterns? I thought they were better than that.
The fact of the matter is amazon is a giant company and cannot thoroughly think through each piece of logic in their system.
The reward for focusing on this before-hand is much lower than just writing a check for $5k to this person and then fixing later (lot of $5k checks from amazon today. Wheres mine?)
>"'They' spin up spot instances which isn't subject to Billing Alerts.
Spot Instances aren't subject to AWS Billing Alerts? Is this common knowledge?
I'm not sure it's true.
Why the cliffhanger? Why not just finish your thought? As is, this story provides nothing other than showing the Amazon provided some helpful customer service.
We had an almost identical event last Sunday in one of our dev accounts: multiple high end spot instances in multiple regions with a newly created security group pointing to a suspect IP.
We caught and corrected it quickly, but we still don't know how the keys leaked out - we have chalked it up to lower security practices since it's not a production account and is shared by more people (e.g. no 2-factor on it). We started to investigate, but then Heartbleed happened.
I wish there were more mechanism in AWS to prevent bills from mounting up, but the basic billing alarms worked in this case. I can't imagine how or why spot instances would be excluded from alerts, their cost certainly is included in the estimates that alerts are based on.
Another reminder to be extra careful about checking in AWS credentials into version control. You never know when you might open source that repository and someone can easily yank it from Github.
Actually this is what happened to us... our AWS credentials had accidentally been put into Github for a hackathon project several months ago.
Coincidentally, the incident also occurred around the same time (April 1-2). We were hit with $13,000 worth of EC2 usage before we shut them down and changed our AWS key... We reported to Amazon, and they are working on a refund.
The instance was spun up on April 2, but Heartbleed wasn't disclosed for almost a week later. I highly doubt anybody used the Heartbleed 0-day to access your account.
According to Cloudflare (http://blog.cloudflare.com/answering-the-critical-question-c...), exploiting heartbleed may actually be very difficult. So yeah, it's very unlikely for that to have happened.
Well, getting an SSL private key is difficult as they don't often get into memory and are quite long (difficult to get from 64k at a time). Whereas AWS credential keys are something that get into your servers RAM much more frequently and are shorter strings. So it could easily be remote memory exploitation. But more likely social engineering or some other easy path in.
People accidentally send those keys off to Github all the time. I'd suspect that sort of thing.
Heartbleed only exposed SSL memory (like incoming or outcoming connections), but not other memory (particularly not program memory), containing AWS keys.
A similar thing happened to me when I made public a previously private repo on Amazon and forgot to scrub it for AWS key. Luckily I got an email from Amazon within 24 hours of the instances being started, and as such my bill was just $360, which in any case they waived off. You might want to check your Github repos.
I have 2-factor auth enabled on my AWS login - but am I right in thinking that if someone has my API keys that they don't need the 2nd factor?
Yes, someone with your access and secret keys can spin up instances, create buckets, and do everything else that the stolen keys are authorized for.
Which is why most things should be done with IAM keys specifically locked down to minimal privileges.
Apparently I have access keys that predate the release of IAM ! Fortunately there's a convenient "disable" link on security keys page.
Agreed
The fact that the consultant - unprompted - knows exactly what is going on suggests to me that Amazon needs to fix this. If one customer gets burned then the customer is an idiot. If lots of customers get burned then maybe its not the customers that are the problem.
Is there anything you have done that might have compromised your account. Moreover bill could as well been much larger than $5000 i guess.
Is it too pedantic to want the tech support to just say they were probably mining cryptocurrency instead of bitcoin? It's most likely the intruder was mining either litecoin or whatever coin is most profitable for the month. I know it's all very much the same but they were almost certainly not mining bitcoin.
A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be worth the time logging into someones account.