Ask HN: What is it with the recent DDoS attacks on Linode?
Linode has been experiencing a wave of DDoS attacks in the last few days. I've noted connectivity issues at Newark and London datacenters. DDoS isn't something of a novelty, but being a Linode customer for several years now I don't recall having that much of a problem with availability before. Is this attack of any special significance?
http://status.linode.com/ [Just posted to status.linode.com] In progress - An update from Linode about the recent DDoS attacks
Dec 31, 22:45 UTC
Scheduled - I’d like to share some updates about the recent DDoS attacks. I am one of several network engineers at Linode who have been working around the clock on DDoS mitigation. While things are stable, I would like to take a moment to publicly address the large and frequent DDoS attacks that we have been receiving since Christmas Day. It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business. The following is a partial list of attacks we have received in no particular order: - Multiple volumetric attacks simultaneously directed toward all of our authoritative nameservers, causing DNS hosting outages - Multiple volumetric attacks simultaneously directed toward all of our public-facing websites, causing Linode Manager outages - Layer 7 (“400 bad request”) attacks toward our web and application servers, causing Linode Manager outages - Large volumetric attacks toward our colocation provider’s upstream interconnection points, overwhelming the router control planes and causing significant congestion/packet loss - Large volumetric attacks toward Linode network infrastructure, overwhelming the router control planes and causing significant congestion/packet loss All of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change. As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. We are working extremely closely with all of our technical partners, including our network equipment vendors and our colocation providers, to prevent future attacks. Once these attacks stop, we plan to share a complete technical explanation about what has been happening. Additionally, we will be announcing the details of an ongoing project to significantly improve our internet connectivity and resiliency. We would like to apologize for the lack of detail in some of our recent status-page updates. Please know that we are dedicating all resources from multiple departments to stopping these attacks. We acknowledge the amount of downtime we’ve been experiencing is completely unacceptable, and we appreciate the understanding and support we have received over the past several days. We will share more information as our investigation continues. Alex Forster
Network Engineer at Linode It's becoming a major issue for us, if it's not resolved by Saturday we are pushing the big red button, we can ride out a couple more days because most of the stuff we write/run for clients is LoB for SME's and everyone is shutdown for the holidays pretty much, after that not so much and it's going to be grim. Why put all your eggs in one basket anyway? Frankly? Because its worked fine til now, each client has their own VPS and linode has been very reliable for 6 years for me and 3 years since I started company. But yeah I should have and did know better :(. What is the big red button? Moving. We did it today. It's been a week. I'm tired of getting texts that my servers are down when there is nothing I can do about it. You and me both my friend, who did you move to? Do you have cattle or pets? Moving cattle should mostly be a DNS flip. Moving pets is harder... and a good reason to avoid pets ;) How is moving to another provider just a DNS flip? It's a huge pain no matter how it was designed. How can you move 100s of GBs of data when your servers are being DDOSed to hell multiple times per day? Well the people here are on Linode, which as far as I know has no concept of S3 type storage.. so they don't have "100s of GB of data", more likely some apps and maybe a database or two that should be backed up offsite anyway (perhaps to S3 itself). If making your environment is running ansible-playbook bootstrap.yml, life is not that terrible to move. We moved to AWS. It's now less expensive than Linode when uptime is factored in. Yeah I'm leaning that way, largely because while AWS isn't perfect it does at least have the "It's what everyone else uses" factor in my defence, that and when AWS breaks half the internet breaks with it so users tend to be more understanding. The attack is clearly of significance though Linode has not made much information public. Many have been told that the attack is bigger and better organized than the typical DDOS attack. There have been multiple methods of attack, and when one is mitigated they change to a different one. Some have speculated that a competitor may be behind it. Others have speculated that the perpetrators may be demanding a ransom to stop. I've been following the issue closely, and that is all I know. Linode support has been careful to avoid answering questions like: "How long will this last?" and "What are you doing to prevent it?" Same have been a customer virtually from the start, all my important stuff is with AWS these days but there are still a few minor things at Linode and its getting annoying. They will come back better from this thou. Its sad to see their business going partly down the drain, will be a huge hit. If I had anything of use at Linode right now then I would be long gone, not many businesses could afford a week like this.