 |
| $17,000 AWS bill in the making. |
4/15/2014 Update: This story was picked up by ReadWrite.
Part 2 Here
The problem with the Heartbleed bug is you never know when and where you'll get hit. Actually, this is true for all security breaches.
Yesterday, I received an e-mail from Amazon asking me to update my credit card info for one of my personal Amazon Web Services (AWS) accounts. I logged in and saw that my running total for April was over $5,300. My typical monthly bill is less than $6.00 which is about 1,000 times less.
At first I thought it was a mistake. I hadn't fired up any EC2 instances this month and my account had no EC2 instances running in my region. I filled out a billing inquiry request form and selected the "call me now" option. Within a minute my phone rang and I was speaking to an AWS customer service rep.
I explained that I've been an AWS customer since 2007 and I've never seen a billing issue like this.
He said, "We've been seeing more and more of this. Check your spot EC2 instances in other regions and you'll see high end instances running."
Sure enough, he was right. In Tokyo, São Paulo, Sydney, and Singapore I had expensive server instances running.
"Your AWS credentials have been compromised," he said.
How did they get compromised? When did it happen? Was my development machine hacked? It couldn't be my Time Capsule since that's encrypted. Were one of my physical servers hacked? Did I have a backup, sitting on a server, somewhere, that was hacked? Am I about to get stuck with a $5,360 bill?
"'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.
"When did this happen?" I asked.
"Let me look," he replied.
My mind was still racing as I tried to figure out the source of the breach.
"These instances were spun up on April 2," he said.
Very smart; launch the attack early in the month so the victim won't know anything's wrong until they get next month's bill.
"Is this related to Heartbleed?" I asked. It had to be.
"No, it's just a case of your AWS credentials getting compromised," he answered.
He walked me through the steps to secure my account.
"Can you see what 'they' were doing with these spot instances?" I asked.
"No, we can't see inside the instances. But, they're usually mining for Bitcoin," he answered.
Ahh, now that makes sense. Spend $5,000 of someone else's money to mine, say, $1,000 of Bitcoin for yourself. Can't follow that money trail.
"I'm going to send you a questionnaire. Please fill it out describing what happened and, due to the large amount involved, I'll need a manager to review it. But you won't have to pay for what you're not responsible for," he said.
I let out a sigh of relief.
I was still dripping with sweat since I'd just returned from a run when I saw the initial e-mail from Amazon. While I was in the shower it hit me. I know how my AWS credentials were compromised. But I'll need to do a little more research first.
4/15/2014 Update: How did this happened? See Part 2 of this story to find out.
Author: Joe Moreno