Ask HN: Hetzner banned me with no explanation. What can I do?
I have been using a 16 vcpu 32GB RAM 50Euro/month hetzner cloud instance as a remote development server for about 3 weeks now. Have not run anything else on the server, just a vscode server and my code.
This morning I received a notice that all my services with Hetzner have been locked with no explanation and no direction for contacting a human. Below is the full email I received.
What can I do?
--- Dear Mr ______
Unfortunately we have had to lock all services you have with us due to violations of our Terms and Conditions (https://www.hetzner.com/rechtliches/agb/) and/or System Policies (https://www.hetzner.com/rechtliches/system-policies/).
We will not be accepting any more orders from you, and your account will be cancelled to the next possible cancellation date, as per our Terms and Conditions.
This decision is final and cannot be appealed.
Kind regards
Your Hetzner Online Team I don't know anything about Hetzner, but there's something completely inhuman about the overall tone of that email even on top of the overall crappiness of the situation. > No lead time (not even a paltry 24 hours) to find an alternative service provider. > No mention or clarification that any data will be available for download in some fashion. Imagine relying on them for anything truly critical and being insta-banned. You can't operate a tech company on a server like that. > No apology for having to do this. Using the word "unfortunately" doesn't count. They're giving a human being a shitty day: the least they can do is playact at being a tad sympathetic. > No explanation of any wrongdoing or a reason for why this is needed. Even a simple "Due to legal requirements" or "excess resource usage" might help. > No way to contact anybody if there's any error or outstanding business issues. > A display of real arrogance by using the word "final" and "cannot be appealed" in the message. > Addressing themselves as the "Your Hetzner Online Team" rather than a specific individual. If a human made a decision, they should own responsibility for it. If a human didn't make that decision and it was some algorithm, there's no way it shouldn't be appealable. They are learning "best practices" from the best (Google and others), I imagine. Going to say this was not the first warning. They would not do this even if you are doing something very wrong the first incident; maybe your server was hacked etc. There is more to the story. I would call them myself as I also do with OVH when they send me panicky emails, and it is always something we can easily resolve. And yes, I did have a few hacks long ago with them that were my fault and port scanners & spambots were installed. But I was told that and got amble time, in rescue mode, to fix and update it all. Unfortunately Hetzner cutting you off seems to be a common occurrence. Even if you haven’t done anything wrong, they can essentially fire you as a customer if they decide you’re too expensive for them. I have used OVH in the past, but these days I would definitely think twice before using budget providers like these. Maybe they treat me better because I am a long term customer. That could be. I don’t think these providers are that budget really; OVH has a budget provider (kimsufi) but ovh isn’t one anymore, imho. Unless compared with cloud and ‘enterprise’ providers. Anyway maybe it is because I am a long term customer of all of these; I do tend to have a talk upfront what I plan todo and tend to make sure I have a contact there. OVH is too big for that now (I had a contact until about 8 years ago in France), OVH is not. If not aws or other overpriced (for dev/test stuff) cloud provider; what do you use as non-budget provider? Well I'm not 100% sure how it is these days, but IIRC OVH IPs used to have a bad reputation, because their cheap prices attracted spammers. On the other hand, my current e-mail provider uses Hetzner and I haven't had any problems getting my e-mails delivered, so who knows. I currently host my stuff on Uberspace, if I needed a 'real' VM I would probably go with TransIP because I already have my domains there and they are local to my country. I'm just talking from a personal perspective though, I'm not hosting any business servers or anything like that. Admittedly it's also very possible that the reason you hear more about issues with OVH and Hetzner is just because they have more customers. I am indeed also overpaying for a VM and a few domains on TransIP (I am also from NL like you I guess) :) And I had massive issues with them in the past. I had less issues with Hetzner anyway. So also a matter of experience. This was the first email I received, after my welcome email on July 15. This was a dev instance, maybe I was hacked, cannot rule that out. But was not expecting an outright ban with no details on what has happened. Sorry to hear it; it seems very strange. Maybe indeed because the account was new and ‘something’ flagged it. I would try to call them anyway; phone is usually easier in this case; find it hard to believe they would check your info and just hang up. Contrary to most other clouds, Hetzner has a DE & EN support phone number: https://www.hetzner.com/support-center Did you call them?
What did they tell you on the phone? I've been with them since 2004 with currently 100+ servers and cloud instances among my companies. Yes, they employ one trigger-happy young sysadmin, who can be quite stubborn, too. But in all the years, they never took any action completely without reason. Like we might disagree about the weight of my mistake, but there always was one. If I had to guess, you were working on Crypto or used torrents. They insta-ban for some protocols. Also, if you connect to too many unroutable IPs, they will create an "abuse" case and disconnect the offending IP from their network. There must be more to the story... If true, I'd check all configured email addresses. They let you configure different addresses for support/bills etc. and will send warnings only to certain addresses. Hetzner is usually good at revolving issues. If you don't pay a bill, they eventually will block incoming traffic from the web. They are still reachable from inside hetzner network and they will unblock traffic as soon it's paid. If the BSI finds Ports that shouldn't be open to the public, they will forward the mail to you and won't take actions. If you disturb their network due to misconfiguration, they will block you, demand an explanation within 24 or 48 hours and unblock you, if they find it plausible. If you call them with technical issues - in my experience - you typically want to prepare logs, traceroutes etc. because they will know enough to provide guidance on how to resolve it. OP Here. I have a single configured email address on which I received my welcome email on July 15 and "Server Locking" email today. Looking into Hetzner dashboard, it seems they did not delete my instance, just turned it off and banned my IP so I cannot ssh into it.
There is an option to request unblocking which I will request soon and which wants me to answer "What caused this problem?" and "How do you plan to correct this problem and prevent for the future?". This was a development instance: running docker, postgres, SchemaSpy, some service emulators, node, vscode and accessed the services through ssh port-forwarding. It seems there is an "Abuse" incident linked to the blocking of my IP but I only see the incident ID, no additional details. This was a dev instance, I did not think about making it airtight. I do not rule out that someone broke into it and violated their terms (this happens with production systems and I am definitely a worse engineer than people there). If this happened, I am happy they locked it down but I wish they informed their users in these cases: I had git ssh keys and other secrets there which I proactively revoked and more information on the incident would definitely have helped choose the right course of action. Some noisy services can cause bans. I have quite a bit of rep with Hetzner, so they didn't outright nuke me, but I once got an abuse email because I was running an IPFS daemon, and the reference IPFS implementation allows RFC1918 IPs and GCNAT on discovery announcements... so dialing into nowhere a lot upset the router. With the new no-ip-at-all option you can set up a Network and set up an extra instance as NAT as you would with a home network. That should cut down on issues like that. Also, the main reason I did not initially add a lot of info to the story is I do not know what is relevant and what is not because the email I received from them contains even less detail than my post. That is the only thing I really wish changes in the future. Would totally use Hetzner again with that (if I am allowed to, that is) Am I crazy for thinking that its hard to take these posts seriously without ANY indication from the op of what they're up to? I know that obviously there's no obligation to share etc, but I can't help but feel like if they truly weren't up to anything sketchy they would be more forthcoming? > without ANY indication of what they're up to They stated in no uncertain terms they're running a remote VSCode instance and nothing else. I've done something similar back in the day just to have a consistent environment (target was a hacked up, cheap, used Android phone I left at home, but same idea -- remote IDE for one reason or another). These posts aren't that crazy when you think about the level of abuse a service like this probably gets and how few dollars per account are spent from the bulk of users. I've been banned from Digital Ocean before without any crypto mining or heavy workloads or failing to pay bills or hosting porn or any of the other sorts of things you might expect. My best guess is that they flagged the fact I was using a privacy card as an indicator of potential eventual fraud. Or else a new fraud model took into account age and didn't take into account that there was no way for a new account to ever become old enough before being flagged, or some other sort of "data-driven" blunder. No biggie though, there are plenty of operators willing to actually accept my money for their services. Added details here[1]. For clarity, I'd just add that I was also running my backend on the instance which could have been buggy and compromised (cannot rule that out). Was not mining crypto, hosting porn or doing anything else that I'd think is clearly against their terms of service. Not crazy but perhaps leaning towards a kind of technological just-world hypothesis. Anything bad has to be because the user, usually the person with the least power and information in the system, did something wrong. The many other humans behind the technology -- the coders, architects, testers, managers, executives, lawyers, 3rd parties -- can never go wrong. They must have studied the entire set of probable situations and devised just and fair solutions to every situation. Therefore, they should not be held accountable though they typically have both more power and more information. They're incapable of making mistakes. “Can someone help me understand why I was banned. I was running a crypto coin test net on my instance for development purposes funded only with a visa gift card. I left all the ports open and used a default ssh password. What happened?” That’s how I read all of these when people don’t add more details. Have you tried this yet? (From the "Legal" section of the Hetzner website) > Online Dispute Resolution in accordance with Art. 14, para 1 of the EU Online Dispute Resolution Regulations Online dispute resolution in accordance with Article 14, Paragraph 1 of the ODR-VO (Online Dispute Resolution Regulations): The European Commission has established a platform for online dispute resolution (ODR). You can visit the platform at http://ec.europa.eu/consumers/odr. Find a different service. The lack of info is lazy and insulting. >This decision is final and cannot be appealed. This translates directly to "FU! We do not want your business." well yes.. they don't want your business. We all hate this... but if you put more details into your HN posting it would be a more effective complaint. Right now, none of us know what your code was doing. Portscanning the entire internet? Botnet C&C? Got hacked because something that was forseeably your fault? Put some details in so that your complaint and theirs don't have the same amount of evidence. I am planning to launch our next product on Hetzner. Now I am super afraid. Any advice? Things I've seen happen the last decade or so: * A data centre got flooded * Another data centre caught fire * One provider went bankrupt * One provider discontinued the product with a week's notice * Another provider terminated the account because they thought the account was fraudulent You cannot build your business based on the assumption that one provider will always be there for you and at the price level you are comfortable with. At some point somehing will happen, and when it does you're screwed if you don't have a plan for switching providers. Besides what was said; make sure you are not doing fishy stuff. I cannot imagine it was as innocent as written by OP. Hetzner doesn’t just do this; even if you would be running some weirdness, they first tell you and ask what it is. Just like OVH does by the way. Maybe you were hacked etc. Without more info this can be anything and everything and is no cause for concern at all. I think it helps if your Hetzner account is at least a couple of months old, with usage. Quite often these things happen to new accounts. Lock your server down and monitor resource and bandwidth usage. You don't want to be banned because someone else is abusing your server. I'm on Hetzner too but will consider moving my hot spare out of Hetzner to a different provider. This might be part of the reason, if OP's account was actually compromised, they signed up this month and got their first abuse report less than half a month after, the type of customer hetzner does not want in general... Still build on Hetzner. Just build up your architecture without relying on cloud specific services or APIs if possible and once a quarter check if you can rebuild the service on another cloud provider Go for physical machines - they're beefier than the cloud instances (RAM, CPU, storage), and cheaper if you architect your application/service smartly This is why all my services use multiple hosting providers. Use OVH What did you run. Facebook, Twitter, telegram(mqtt) or docker copyrighted code on Hetzner? Usually, code from those companies trigger environment variables checks and take- downs. Wait what? How does that work? Is this documented anywhere? Ran docker and one of my endpoints inserts rows to a google sheets via its API... What kind of space crack are you smoking? What? That is interesting No, it's not, because it's absurd bullshit that makes absolutely zero sense if you think about it from one of at least the different directions. I'm sorry you had such an adverse reply from so many here. Your post seemed pretty clear about how you were using your server, which is to say "not much that should have drawn any attention." There have been a lot of similar reports about Hetzner competitors, so it seems one just has to maintain off-site backups and be prepared to randomly jump ship. There are lots of reports of this in the DigitalOcean sub-reddit. As to the cause, I've gotten caught up in things like this before... no so much from cloud providers but from other e-commerce vendors and even on-line banks. I've had some luck writing paper letters, not going away without an answer on Twitter, and filing government complaints. The general gist is that like like spam is a problem for email, other types of fraud are a problem for cloud providers and merchants. They're turning to some of the same kinds of tools that are used against spam... with the same mediocre results. I've taken a lot of time to get under their skin and get to the root cause. I've been successful about half the time, and the reasons are usually lackluster: - You used a VPN when you signed up years ago
- The bank the issued your credit card (the first 8 digits) matches a lot of other fraud events (this is particularly the case with gift cards, over the counter debit cards, and virtual cards... though I've had the same problem with major brick and mortar banks.)
- You had account activity that doesn't match normal hours for your time zone.
- I ran an ad blocker, which also messed up some CAPTCHA/JavaScript thing
- I have "load images" disabled on my email client, so it looked like I wasn't opening mail from them.
- Other fraud occurred from a similar IP address. Often they use plugins from commercial anti-fraud companies, much like Facebook or Google ad plug-ins. These companies look at information from lots of places and try to identify patterns among accounts that later are reported as fraudulent. We use one of them where I work. It's about as effective as a spam filter, meaning it catches most but has both false positives and false negatives. You can tune it to be more or less aggressive. Depending on where you are in the world, you may have more rights to dig into it than Americans do. Also, if you used a promo code, you might ping the advertiser and let them know as this hurts their brand as well. I hope this helps. Was there anything going on in your code? I'm not sure hetzners policies but for example if your code is utilizing certain ports and traffic types that they might have limits on? The response from them is very flippant and robotic though. It may be an automated action but I'd be curious to hear your experience with the "human" you get in touch with. Edit: as for the decision being final this is usually just to deter bad actors. I've had some issues with a compromised server when colocating who said the same. This was a pain to prove, they did overturn it but I imagine it had something to do with the higher fees being paid to them. Next thing in the EU must be legislation where the company says whay you did to violate which term. Paypal closed a account of mine (business) while keeping another (private). Amazon closed several tries to sell there, only worked with an incorporated company to sell the book of my wife. Without Amazon you can't sell a book (fiction, no massive social media following). Both companies of course, no mention of the reason, just a link to their TOS and this vague speak. On top of that we need a way to applay to an external arbirter for companies that have more than 10% market share. They also banned me because I forgot to pay one month ( and my contact email was one that I didn’t use anymore ). After a year I tried to resubscribe and they denied me, ( I offered to pay any debt , but they refuse ) Hetzner never banned me but they demanded a lot of info at sign-up like a photo of my ID which contains sensitive info like my social security number. Can be used for identity theft. I blanked that out and they didn't accept it until I showed them the national police website where they showed how to do this and recommended to always do that. I only used them for a while, eventually I mixed to scaleway which was cheaper and doesn't need any invasive info. I've been a happy customer there for years. I even run an IRC bouncer there without any issues, which many such providers specifically forbid (eg OVH) curious - what kind of ID would have your SSN on it?