Ask HN: My VPS got hacked and now I'm facing a massive bill. What can I do?
I've got a VPS which I use for small programming projects and college assignments. Two weeks ago I received an e-mail from my provider, stating that "your VPS has been transmitting a lot of outgoing traffic which results in a very large traffic usage bill". In September on my 500 GB data-limit VPS, it had been transmitting 27 TB of data traffic. This resulted in a € 3300 extra charge on my € 15 VPS. I'm expecting a similar bill for this month.
Of course I immediately shut down my VPS after the notice two weeks ago, but by then it had been using these amounts of traffic for a month and a half.
What are my options here? I can't afford to pay > € 5000 unfortunately. Does anyone have similar experiences? Incidentally, since many HNers probably come at this from a mental model of "Anything which appears on an invoice is non-negotiable and simply must be paid": a B2B service provider which collects payment after services are rendered is knowingly taking on credit risk and has already priced non-collectability of some accounts into their services. You may be overestimating how much drama is required for someone at their company to say "Wow, really? OK, sorry about that. I'll write it off." This is one of many, many, many reasons why we don't generally do cost-based pricing and, when we do do cost-based pricing, the markup is absolutely phenomenal. It has to include risk premiums. As long as it do include risk premiums, you don't have to sweat the small stuff like e.g. an uncollectable $4k invoice. (n.b. Small stuff! $4k hiccups are utterly routine events and largely dealt with by processes rather than by treating them as sudden emergencies, even if they feel like that to natural humans.) Also OP, if they don't write it off make sure to tell us all who it was so we can avoid them. The same thing happened to me with amazon. Amazon pid for it. It's highly unreasonable in my opinion to ask the customer of a VPS to pay for damages caused by a malicious attacker. It's tantamount to a landlord expecting you to pay after an arsonist comes along and burns down your apartment, just because you happened to be renting it at the time. May I propose another analogy: - a landlord* expecting you to pay after a squatter came along and opened a faucet in the basement (where OP rarely goes) to fill its own super tanker * or the water company ? I have had this experience. In my case the bill was altered to $0 before I'd even got to the point in the conversation where I intended to ask for some relief. It most certainly doesn't cost them that much, and it's within their interests to keep customers coming back for repeat payments than soured by being forced to pay an exorbitant fee for something which was obviously not their fault. Yeah, the $4k is kind of a manufactured figure, bandwidth expenses are particularly inflated, but it makes sense at smaller scale - if you have a $15/mo plan with 2TB of transfer and you go over 1TB, a few bucks penalty doesn't sound outrageous, and their cost is impacted by higher network management costs. But, if you have a huge spike that wasn't really your fault, it doesn't cost them any more to write that off than it does the bandwidth that is consumed by a DDoS attack that is mitigated by a firewall. When I worked at Rackspace, we hosted a very popular flash cartoon for one month after yahoo kicked them off the $5 hosting plan for pushing god knows how much bandwidth. They basically saturated a gigabit network port from a single server, and we sent them a bill for like a gajillion dollars. They went to another hosting company, of course, and I think got the bill down to something they managed to pay off. Someone obviously had porsche-eyes, I thought it was kind of a shitty thing to do to the guys, who came to the office to make the voices of their characters for us and stuff. I worked for a hosting company and we'd normally charge time spent handling the issue plus the upstream cost of bandwidth, depending on how nice the customer acted. 1. Report the incident to the police. Right now. 2. Report it to the VPS provider. Explain that you've reported it to the police. Ask for their cooperation in investigating the problem. You do not have to pay. If they try to force you to pay, depending on your country, you'll probably end up in small claims court where you'll find judges are very reasonable people who usually side with the little guy. (IANAL) Yes. This happened to a company I worked for in the UK in the last 1990s. We had a half rack full of NT4 machines and someone got into our kit and used it to run a pr0n FTP. They took us to court to pay up and the magistrate said we had no bill to pay and that it was a waste of court time as it wasn't intentional. You're right about reporting a crime though even if the police don't take it seriously. A crime ref number goes a long way on its own. Edit: we had to move our kit sharpish though as the company exercised their right to throw it on the street within 24 hours. MAKE SURE you file a police report , and are diligent in recording and logging everything and have a long paper trail. Including ALL customer support interaction , police logs, diary , journal, etc. Judges will show favor if you have a paper trail. Not going to lie I think that the chance of this getting to a judge is about 0.009% I hope cyber crime police has improved their working in recent years. 5-6 years ago a Ebay seller defrauded me. I filed a complaint with all the information I had since everything is online and involved bank transactions there had to be a money trail. I never heard back from cyber crime cell. I don't know how many folks were defrauded by that person before and after the incident with me. Doubtful. I got a call from someone about a month after my GF's iPhone was stolen. He'd bought it on eBay and asked if I'd got a new one yet - her phone gave a number to call and was activation locked. He wanted to know so maybe I'd give the password so "he wouldn't be ripped off, too". I'll give him credit for having the balls to ask. But (as I later found out) he knew he was buying an "activation locked" iPhone. I said a few things like how do I know you're not the thief, etc. He pointed me to the eBay listing which, sure enough even had the IMEI with two digits transposed (plausible deniability, I'm sure). He contacted the seller and said "Tell me why I shouldn't give this phone to its rightful owner and then file a fraud complaint with eBay and get a refund?" Unsurprisingly the seller offered to take the phone back. So he sent it (I didn't care, and while I knew the insurance company was about as unlikely to care as the police were, I didn't want to do anything that might trigger insurance fraud questions - "This phone was reported stolen, unlocked using your credentials. Explain.") and got me the seller's home address. I contacted the insurance company. They didn't care, just told me to file a police report and send them the case number. Looked at the seller's profile, quite possibly the sketchiest thing I've ever seen. Bunches of phones, all "activation locked, no charger". Tablets, no charger. Laptops, no accessories or charger. At least 50 or so. Gave that info to my local PD. Their response, "We won't investigate. He probably bought it from someone and is selling them. Could have gone through a few people first." I didn't want the original thief caught but this guy was openly selling stolen gear. Hell, the message on my GFs phone said "I don't care about stealing the phone. Will trade cash for it.". They weren't interested. Bear in mind, this isn't someone complaining about their car at the impound lot in LA, a la Big Lebowski, this is town of about 40,000 with a well-funded PD (I work for Fire in the same town). The urge to drive to this jokers house in the middle of the night and pour sugar in his gas tank was one I avoided, but only just. I doubt it has improved much - in general if I report a crime to the police I don't expect it to be solved. The reason for reporting is that it is the first step in initiating most civil means of compensation (insurance, company cancelling the charge, etc.). I wonder what experiences we have with police in similar situations in recent years. In the USA many years ago I brought a telephone bill dispute to police. They explained, not without kindness and patience, that the legal question involved intent. Consequently it wasn't a matter where they could easily provide assistance. What was the situation? In the OP's case, they're alleging a very definite crime of illegally accessing a VPN to do nasty things with it. I think at least taking a report is going to be standard behavior, though I wouldn't be surprised if they just let it "sit on the shelf". If it's just a garden variety dispute with a company, then it's more likely to be a) a civil matter, and b) subject to interpretation. Well, that's the point. The specifics make a difference. In my example, it appeared that a roommate unknowingly permitted a friend of theirs that I never met then or since to run up big charges at billable services on the apartment telephone line that was in my name. (I understand what my mistake was and because it was so long ago it's not a sore issue.) I've been in the exact same situation with AWS ( http://cl.ly/SHOu ). It was a nerve recking couple of days but I contacted AWS support and they were extremely good. They helped me secure my machine and then cancelled the 1.4K payment they were going to take from my account. In all the whole process took 2.5 weeks and I only had to pay $15 for the I/O requests. The best thing I can recommend is to talk to your host and tell them honestly you can't pay that much and you weren't the cause of the charges either. AWS also has billing alarms. If you're running an account which should cost $15/month then for the love of god set an alarms at e.g. $20/month. I had a reserved instance for 12 months, forgot to renew it, and on the 13th month (when it was on-demand) the usage creeped over my cap and I started getting alarms allowing me to kill the instance, renew my reserved, and restart it. Saved me at least $10. On a related topic, I wish VPS providers allowed you to pre-pay. With Microsoft's Azure I have an MSDN Ultimate account, which has $150 of pre-paid credit on Azure. When you go over the $150 they just shut your stuff down rather than charging you (in fact I don't have a CC on there at all). They don't even offer this kind of service to non-MSDN subscriptions which sucks, I'd love to just pre-pay $50/month to them and have everything shut off when I exceed it (so it becomes a "no risk" playground). You can prepay on Digital Ocean. That isn't what their FAQ claims: > Can I pay in advance? > We do not currently accept pre-payment, but you can add additional credit to your account at anytime by sending in a PayPal payment. Actually you can not pay a vps in advance, but it is like the "prepaid" mobile phone system... You can add credit to your account and pay your bills with that credit, which I think you could call "prepaid". I've seen AWS credit charges that people intended to make but didn't want. We have some public AMIs that include charges for our product. Twice AWS support contact us to ask if we would credit someone back on the order of $500 out of our pocket because their customer didn't realize running an AMI would incur charges. Yeah, I was pretty surprised by this, too. It seems AWS gives people one freebie, though. Did you grant the credit? Why or why not? We did because AWS did and we didn't want to look bad. In aws, you can set up billing alerts, so they will email you if you go over X per month. It's a good idea to set that up, so at least you'll be alerted as soon as possible if you get hacked. This is how you handle it. Billing alarms at the highest priority. I expect mine to fire off around the 20th of each month. I expect the second alarm to never fire. Hacked account => billing alarms get turned off. So you won't find out. Yes, it's not foolproof, but you might get hacked by someone who forgets. It's another lawer of protection. Same here, not as big as 4k, but only $200 more. AWS took care in an awesome way. I work for a VPS provider in the US. These situations are common and we usually just issue a credit and give a reminder to the customer to please secure their server That brings me to my point. How did the hack occur? When you get a VPS you are fully responsible for what goes on in there. It is your responsibility to secure it and keep it updated. It's not the provider's fault you did not apply the latest security updates. It's not the provider's fault your Java application was using outdated and vulnerable libraries nor is it their fault you didn't set a CAPTCHA in front of your submission forms. Either hire a competent sysadmin if you can't take care of that yourself or find a provider that offeres managed hosting instead of a VPS, as that's what you'd most likely need. There are some cases where it's the provider's fault such as the Linode BitCoin hack a few years back but mostly it's just poor server maintenance People that rent a $15/year VPS use it to run an IRC bouncer or a small web log, something you don't need to know a whole lot of sysadmin stuff for. They just need a machine that's always on. It's hardly worth hiring a sysadmin for (I find that suggestion laughable, to be frank). Managed hosting doesn't allow you to do much else besides hosting a website in PHP, which is not enough for plenty of use-cases, including OPs. > $15/year VPS Do tell, where do I get one of those? Cheapest I know of is $60 ($5 a month). Here's one that I know of: http://ramnode.com/vps.php Other than that, I'm sure you can find something on webhostingtalk forums. Check http://lowendbox.com/ , though YMMV. http://serverbear.com/compare/vps is a great place to find these VPSs, they list some of the smaller services. I rent one on BuyVM (http://buyvm.net/). I use it to run an IRC bouncer and an IRC bot. cloudatcost has $1/mo small servers, or you can buy one 'permanently' for $35. I work for a company that provides VPSes. In a situation like this, they can see the usage is aberrant and they can see it's not normal based on past bills. They'd likely offer a large credit if you say you didn't intend to do this, and it doesn't look like a fraudulent account. That being said, they themselves probably have bandwidth costs, and are not at all likely to forget all of the charge, perhaps half at best. Stop using providers which charge a ridiculous price for bandwidth (like AWS). There are many excellent alternatives where a TB costs only a few dollars/euros. like? Hetzner has the first 20 TB free and then charges 2 euros per TB. Any sane provider with no DDoS protection will nullroute you on incoming DDoS, that's not Hetzner specific. If you expect to get DDoSed, buy protection or go with OVH. The problem wasn't the nullrouting, it was 'contact support to re-enable... and support isn't open until Monday'. Kimsufi (OVH) if you need really really cheap and beefy dedicated servers, with pretty much no support. They are great for hobby projects. Linode, DigitalOcean, OVH, Hetzner, and so on... 1and1
leaseweb
ovh I would begin by contacting your VPS provider, explaining the circumstances which caused the bill, and asking "What are our options?" Have you talked to your VPS provider? They should be able to cut you a break; after all, that 40TB of traffic cost them only a small fraction of what they're charging you, so if they're reasonable you should at least be able to get them to reduce the charges to their actual cost. You might also offer to suggest writing up a post mortem for them, that they can provide to their customers as a lesson/tutorial on how to protect a VPS. Finally, you can suggest that they might want to implement (and perhaps help them implement it) some kind of warning system, i.e., if a VPS suddenly begins using exorbitant amounts of bandwidth, and far more bandwidth than it ever has before, they really should email/text the owner an alert within 24 hours — not let it go on for 6 weeks. I'm surprised that they don't cap/throttle the bandwidth once you go over your plan's limit, to go along with sending you alerts. It borders on negligence on their part that they don't already have such a system in place. In my opinion, it is negligence to an extent that OP should not have to pay for this, and he should find a new VPS provider. Why? He failed to secure his server. Why is that the fault of the VPS Provider? Depends on your provider. Amazon AWS is known to have waived such bills in the past, see for example http://readwrite.com/2014/04/15/amazon-web-services-hack-bit... To prevent such incidents Linode have alerts of traffic/cpu/disk thresholds. For example you can configure notification if your bandwidth utilization more than N Mbit/s in duration more than N minutes. Very useful for DDoS prevention. And look at your dashboards once in a while. I'd find it unusual if I saw my toy VPS cranking out 100mbs for a week straight! Make sure you tune those alerts appropriately and know when your systems should be doing things that will trip them. We've had a couple of boxes that regularly trip cpu warnings during normal operation, but only inside specific time windows. Knowing what's not normal is vitally important with this stuff. I had something similar happen with AWS but the bill wasn't as high since they ended up flagging my box as spam-producing and shut off all outbound traffic. I'd just ask them and see if they can remove the charges, it worked in my case. I had the same thing happen to me. I wrote about it on my blog http://mattarkin.com/protect-your-azure-linux-vm-aka-how-to-.... Basically I complained to Microsoft, they said they'd waive the charge but since it was for a linux vm they said they couldn't cover it. Then I complained to American Express claiming it was an unauthorized and fraudulent charge. Amex sent the dispute to Microsoft and they never responded so I wont the chargeback. I can understand how that could happen and what a problem it would be. I had an experience with a telephone bill myself, but the story is not going to help you. I would suppose your first and best resort is to consult your lawyer, advocate, solicitor, barrister, Anwalt. I wonder what your relevant legal jurisdiction is. I wonder whether it would help if you can account for your own whereabouts and your own usage of endpoint data services. I wonder if your method of payment to your VPS provider is mediated by a financial service that can help you dispute the bill. I am not a lawyer. I assume you are in europe. I'd suggest simply talking with your provider, explaining the issue and asking them to investigate. I honestly expect them to cooperate and be understanding. If they insist for you to pay: simply don't. State the truth: You can't afford it. Tell them the only way they will see this money is by taking legal action against you and even in that case you won't be able to comply - as you don't have the money. Hope it helps :( Just in addition to some other helpful comments: based on posting I assume that your are Dutch or Belgian, located in Europe and are buying this VPS as a private consumer, not a company. Which means your case is probably covered by consumer protection rules when it comes to informing you about data usage, and I seriously doubt a VPS provider has covered their ass as well as mobile providers tend to do. Anyone have tips on how to secure their Linux VPS? I just set one up and disabled SSH password login, locked down all the ports with iptables (using ufw), and enabled fail2ban. Anything else I should install or configure to make myself a little more secure? Was considering tripwire but I dunno how much a headache it would be with false positives as I change things on the server. A very common attack vector is through installed web applications. Especially if you run wordpress with a lot of plugins installed, be sure to enable correct read/write settings for /var/www, and update your application frequently. Malicious entities runs 24/7 scans towards indexed URL's attempting to exploit various vulnerabilities, and many of the vulnerabilities allows remote code execution, upload of php files etc. This can be used to upload malicious code, simple php-webshells, and then your VPS is suddenly a part of a DDoS/Scanning network. Exploited Wordpress sites are a problem, Zeus/Zbot-Trojan is often seen downloading updates/configs from these, and they are also often used to redirect users to Exploit Kits. I came across these two tools recently that seem interesting:
http://www.rfxn.com/projects/linux-malware-detect/
https://github.com/emposha/PHP-Shell-Detector Not installed either yet (LMD could really use some .deb packages) but could be a useful alternative to Tripwire On my personal machines, I also setup a cronjob to automatically install updates. There's a small risk of breakage (I had one in 5 years), but I prefer that over a bill like op's. PSA: Set up billing alerts! You should always have a notification sent to you when your monthly bill exceeds one or more dollar amounts. For example, if you're using AWS, Amazon CloudWatch lets you set an alarm on a billing metric to notify you automatically. Post on WebHostingTalk.com - just do it. You'll get attention from the host, other hosts who will sympathize, and you'll see that they'll just write it off. Post the link when you do and I'll be sure to comment on it (I'm somewhat very-active at WHT) I seem to be missing something. You knew it was happening when you got the first bill, but let it continue for another half month before shutting it down? In addition to the other comments, make it absolutely clear to them (with proof if needed) that you're a student. Make sure Elastic Search is not accessible from a public IP address (this is what likely got you in the mess to begin) Do you know how you got hacked? cut your credit card report it as stolen tell the host that it wasn't u :p I feel really sorry for you situation. I first suggest talking to the hosting provider and explain what happened. Any decent service will give some discount in this case. Unfortunately, I can't think of anything else. I wish it was realistic to tell you to go to the police. Also, if you would give your email, I would definitely consider sending a donation through paypal... Hopefully other readers here will do the same. There is one great reason to go to the police: it establishes a paper trail documenting that a crime was committed. It isn't necessary that the police catch the bad guys for that paper trail to be advantageous. (Examples: police reports make CC disputes and legal declarations much easier and more likely to be given weight as other than self-serving explanations of a deadbeat. It may also trigger insurance policies either for you or for the VPS company.)