Second large Hetzner outage in a week caused by DDoS attack
status.hetzner.comI thought OVH and Hetzner were the source of a ton of these DDoS attacks. Their IP ranges always seem to be in abuse logs.
Cloudflare write in a recent attack:
The top networks included the German provider Hetzner Online GmbH (Autonomous System Number 24940), Azteca Comunicaciones Colombia (ASN 262186), OVH in France (ASN 16276), as well as other cloud providers.
Hetzner operates a 5-10 Tbps network, roughly the same traffic volume as all of Spectrum/Charter Communications (the 2nd largest cable company in the US). They show up everywhere because they are a big part of the internet.
A wise network operator once told me - never shit on people when they are under attack. Because in the not too distant future you are going to be the victim.
Umm.. AWS and GCP and friends dwarf these guys, but I zee Hetzner, OVH and DigitalOcean in these things.
I bet a big factor is that AWS and GCP charge obscene markups for bandwidth. Much harder to get away with.
Of course, no one could afford a ddos from AWS or GCP. Even in the case of compromised machines, the huge traffic bill is going to alert everyone pretty quickly.
I mean that makes sense no? Attacks like that rely on compromised servers so it shouldn't be a big surprise large hosting provides are among the biggest attackers. Other large ISPs like digital ocean and Alibaba are among the top attackers in that attack also.
I assume this attack is UDP based unlike the one you linked too.
Where are the AWS and GCP ranges then?
They aren't even in the top 10 here. Its the claim hetzner is larger than AWS? I find that highly unlikely.
We probably don't see AWS or Azure as source if these DDOS attacks because of egress costs.
People aren't paying with their own money for DDoS machines normally. Well, maybe there are some small operations like that. But often the traffic comes from a hacked service that's a part of a bigger botnet. It may lead to a larger detection ratio on AWS/GCP, but the attackers are not paying the costs.
Attackers aren't paying with their money, but it's important for them to go undetected. If your bandwidth is a fixed cost (ie isn't related to your actual usage) you are much less likely to keep an eye on it than if it costs an arm and a leg.
At their size, don’t they have some kind of hardware-level packet filtering ability like cloudflare to protect against these attacks?
Not if the level of incoming bandwidth exceeds the available bandwidth of the circuits involved.. you can't filter it when the link is saturated. Cloudflare uses other techniques like global distribution so aggregate bandwidth is higher than the attack bandwidth
does anyone else have a network that can do what Cloudflare can do? seems like magic sometimes.
The Netherlands has NaWas non-profit service that filters out DDOS attacks, in Q1’22 7,4 times per days with DDOS traffic up to 300Gbps. It’s a few man shop, costs of membership are low. From their FAQ https://www.nbip.nl/en/nawas/faq/ :
The NaWas infrastructure is designed as an on-demand service. After detecting an attack, the traffic is routed via BGP to the NaWas hardware and then the mitigation process starts. All traffic is then rerouted and the own connections can thus manage with less capacity and thus remain cheaper.
To connect to the NaWas, a port must be available from one of the following parties: AMS-IX, NL-IX, LINX, NET-IX, Top-IX, M-IX, V-IX or one of these cloud interconnects DCSPine, Epsilon, Megaport.
Yes, there's a few distributed DDoS protection services. For example Fastly, Akamai, GcoreLabs, and a few smaller ones. They're mostly less evil too as a bonus.
Perhaps OVH?
This depends on what type of attack it is. If it's volumetric, no amount of packet filtering is going to help you. If it's protocol-level attack then yes, some form of high performance WAF will be helpful if you have the filtering capacity.
Likely the attack isn't an overwhelming volumetric attack as I assume they have some fat pipes and big routers, but there's likely a bottleneck somewhere in their network.
They state using hardware DDoS protection but it seems not to be sufficient: https://www.hetzner.com/unternehmen/ddos-schutz
Compared to OVH their network / DC are very small, also their DDoS protection is inferior.
I am wondering what the attacker's intent is
Retribution :-)
Last time it took about 9 hours: https://status.hetzner.com/incident/129728ce-ba25-49b6-96cc-...
Anyone know if it is possible to mitigate the impact of Hetzner blocking UDP traffic on port 9000+? These outages whacked our Kubernetes clusters (Calico + vxlan + Wireguard). https://serverfault.com/questions/1100482/how-to-limit-udp-p...
Excuse the ignorance, but couldn't ISPs block the attacks?
That would make sense, but its hard:
- You need to identify the traffic to be filtered and the post states: "Due to always different destinations (IPs, ports, packet size) (..)"
- You need to maintain some agreement with a large number of ISPs
- You need to maintain some gossiping infrastructure to these ISPs
- ISPs may not care about your DDoS attack
Yes, network operators (should) participate in centralized black hole services like UTRS[1]. If you can identify the specific IPs that are under attack you make a BGP announcement to other participating networks asking them to drop traffic to that IP within their networks.
As a participant you can avoid paying to send outbound attack traffic, and also identify attack sources within your own network.
>This concerns UDP traffic on port 9000-65535.
Does anybody know what usually runs on those ports?
That's 56,536 different ports. Half of everything (that uses UDP), more or less.
I would expect 95%+ of TCP traffic to run on 22 (ssh), 25(smtp), 53(dns), 80(http), 443(https) plus another handful of lower than 1000 ports. Even common dev ports (3000,5000,8080) are below 9000. I don't think that's much different for UDP. Even most games probably rely on something <10,000.
> I don't think that's much different for UDP.
It is, because of the way that UDP is typically used for different applications than TCP. While there are a few old, well known TCP/UDP pairs like 53, UDP is more often used with a dynamic port assignment scheme sometimes with a coordinating TCP protocol - such as SIP/RTP for VoIP that uses >16k, WebRTC, etc. A lot of games uses ports above 10k. https://help.generationesports.com/hc/en-us/articles/3600611...
I think games.
Hetzner is a popular host for game servers.
Besides games, I think many AV things, including VOIP and perhaps WebRTC (definitely UDP, less sure about port number). Possibly also HTTP/3; the server picks the UDP port number IIUC.
Isn't it most things that aren't a well-known service?
It's interesting that 9000 is the starting port for Ethereum consensus clients, although the participation rate does not seem to be affected.
Source ports of DNS reflection attacks, presumably.
Online games (MMOs, shooters, etc) come to mind
MMOs over UDP?
That's the preferred protocol for ultra-real-time games because a few ms ago is not helpful information to spend time recovering. A sufficiently fast-moving MMO could apply
What MMOs use UDP? Asking sincerely because I have never seen one.
Anything with real time communications like an FPS would use UDP as stale action data is mostly useless. The latest state of is all that matters.
Most such games will either layer their own streaming channel atop UDP for guaranteed ordered delivery of important messages or use a separate TCP socket as well.
all of them, its been like that for a long time. im resurrecting an old mmo, and even then it was UDP.
WoW, FFXIV, ESO, GW2, etc
Some might use TCP for login and character select, but its UDP while playing.
WoW is 100% TCP. idk about the rest.
You must not be looking very hard, pretty much every game engine uses UDP as the network transport. There are some notable exceptions like Java Minecraft.
MMOs often have TCP connections for things like chat and services like auction house (often even HTTP microservices), but most of the gameplay is still UDP.
Absolutely
I fear e.g. wireguard is affected.
[spiderman-pointing-at-spiderman.gif]
seriously, aren't they commonly the SOURCE of many DoS attacks...
any hosting provider where some random person on the internet and $5 of credit on a prepaid visa card will have this problem.
Hetzner requires government ID to open an account
Since when? I've never provided them an ID and neither have they asked later.
At least 3-4 years. That's when I signed up.
There is also the annoying confusion that some attacks involve spoofing the victim's ip to other hosts so the reply goes to the victim while masking the attacker's ip(s).
Maybe, just maybe, rely less on embedded framework on embedded framework that spit JavaScript that gets 95% unused. If for a simple outage apology page the output was 1.7MB, I can only imagine for their normal pages how much it is. At this size I feel only like 10k legit users would unwillingly do the outage anyway. But hey, Kubernetes and Node.js is all the rage nowadays.
"I have only made this letter longer because I have not had the time to make it shorter." - Blaise Pascal