Ask HN: Help mitigating a relentless DDoS attack (4th day) on Onetimesecret.com
Hi HN, I'm Delano, the creator of Onetimesecret.com. We're currently facing a severe DDoS attack that's been ongoing for four days, with traffic reaching up to 4 million packets per second (packets per seconds isn't something we've kept track of, but I'd estimate around 50-100 pps -- to put it a completely different way, we do about 250k+ secrets a month). Our site has been more down than up, and I'm pretty tired at this point from lack of sleep and stress.
Some of you might remember us from over the years. In particular I know it's a common refrain that our encryption sucks. I can say that the reason it hasn't been improved is that our number 1 competitor is folks doing nothing. That is, not using anything at all to protect password and other sensitive info. We are working on improvements though and will get to it, but right now, we urgently need help with DDoS mitigation strategies. We've tried increasing our server capacity and implementing server-level traffic filtering, which has had limited success. The ongoing attack is overwhelming though. And definitely beyond what I would consider manageable with HAProxy ACLs and fail2ban.
This attack is obviously impacting our ability to serve our users. Any advice on immediate actions, recommended services, or long-term strategies would be immensely appreciated. I'm open to all suggestions and willing to provide more details if needed.
Thank you, HN community, for any help you can provide in this tender and critical time. Have you considered engaging the services of a CDN such as Cloudflare that specializes in such mitigations? Rolling anything on your own to mitigate this level of traffic is going to be a large undertaking and not something you're going to be able to do whilst under attack. Since from your post this looks to be a layer 7 attack your options would either be putting your service behind Cloudfront or Cloudflare and using their respective ddos mitigation tools. They also can provide support to get things configured and working effectively. There are other similar solutions out there that I've not had experience with so can't comment on, but utilizing one of the hyperscale services will be your best bet. I would suggest you sign up for Cloudflare. Since you will likely not want to move your authoritative DNS zone to Cloudflare you can do a https://developers.cloudflare.com/dns/zone-setups/partial-se.... This will require the business plan $250 a month. If you can move your zone easily then you might be able to get away with the free account. Once in place, spin up a new load balancer with new IPs for your service or update your current LB but don't publish the new IPs to your DNS zone. Configure Cloudflare to proxy to these. This will keep them hidden from whoever is attacking you. Within the Cloudflare site config, under Security -> Settings -> Security Level set this to "I'm Under Attack" and this will start to present a challenge page to all users to confirm they're human before it forwards the traffic on to your origin servers. That should take some pressure off and will allow your legit users to still gain access to the site. Thanks, this is a big help. There's a bit more info on our blog: https://docs.onetimesecret.com/blog/2024-09-12-ddos-day-4 https://docs.onetimesecret.com/blog/2024-09-09-denial-of-ser... Where are your servers hosted? At Hetzner, in one of the data centers in Germany.