Hetzner outage - UDP forwarding affected
statusgator.comThere was no UDP connectivity from any of our servers for ~10 minutes, but TCP was working fine, I could SSH into them.
Even Hetzner's Robot interface was down.
Edit: https://status.hetzner.com/incident/5e901309-0873-42af-90c5-...
Very interesting. That would suggest to me that all traffic is passing through a stateful device vs. direct connectivity from a router. Routers can give simple bandwidth metrics. A stateful device or mirrored ports to a netflow analyzer would be required to create detailed connection logs though mirrored ports wont break state such as UDP virtual tables being full. Are they doing detailed logging of traffic?
> We are currently experiencing a fault with UDP forwarding.
Exactly right.
From our logs, the main outage went from 21:01 to 21:10 UTC.
EDIT: Also https://status.hetzner.com/incident/affecfea-fd04-41a1-a680-... (Fault report on Cloud API and Hetzner Console)
It seems to be back up. I can access https://console.hetzner.com/ now, and our VPS are back online.
Yeah first time in 6 years being with them that I've had an outage of this scale. Cloud VMs and Dedicated servers all down / intermittent
Everything went down but seems to have recovered now.
They sent out an email about raising their prices earlier today. Perhaps it was everyone running for the doors.