Settings

Theme

Parts of AWS are down

33 points by Bob312371 6 years ago · 19 comments · 1 min read


RDS & S3 is down for me right now in US-East region and apparently some parts of Frankfurt are also down.

There own favicon is timing out: https://awsmedia.s3.amazonaws.com/favicon.ico

lflux 6 years ago

DDos against amazon's DNS servers. One mitigation is to use regional endpoints for bucket hosting if your website relies on serving assets - i e foo-bucket.s3.us-west-2.amazonaws.com instead foo-bucket.s3.amazonaws.com

  • champbronc2 6 years ago

    Where you see this is DDoS?

    • jared52 6 years ago

      I just got off a chat with an AWS agent:

      "We are investigating reports of occasional DNS resolution errors. The AWS DNS servers are currently under a DDoS attack. Our DDoS mitigations are absorbing the vast majority of this traffic, but these mitigations are also flagging some legitimate customer queries at this time. We are actively working on additional mitigations, as well as tracking down the source of the attack to shut it down. Amazon S3 customers experiencing impact from this event can update the configuration of their clients accessing S3 to specify the specific region that their bucket is in when making requests to mitigate impact. For example, instead of "mybucket.s3.amazonaws.com" a customer would instead specify "mybucket.s3.us-west-2.amazonaws.com" for their bucket in the US-WEST-2 region. If you are using the AWS SDK, you can specify the region as part of the configuration of the Amazon S3 client to make sure your requests use this region-specific endpoint name."

    • smudgymcscmudge 6 years ago

      I'm not OP, but I heard that unofficially from somebody who works at aws.

    • lflux 6 years ago

      From the phonecall with our AWS TAM and support reps

joshuapinter 6 years ago

We manually updated our `/etc/hosts` file with the static ip for our S3 bucket(s) and that fixed the issue temporarily.

nakodari 6 years ago

Glad to hear the issue is on Amazon's side. Our team panicked and tried to find every way to resolve this thinking it was possibly a misconfiguration at our end. I hope it's resolved soon. We're receiving complains from many users who are unable to upload their work due to this issue, I will let them know Amazon is on it.

  • Bob312371OP 6 years ago

    Same here. At the time the status check on AWS was not reporting any issues and there was nothing written about it online. As a result I assumed it was an issue with my account.

    I only confirmed it was on their end when I noticed the favicon on the AWS console wasn't loading.

    • ej12n 6 years ago

      Same here, I was running around like a headless chicken and wondering what I was doing wrong when elasticbeanstalk just wouldn't deploy anything. The fact that we doubt ourselves rather than thinking aws is down is a good damn indicator how reliable they are, luckily it happened during daytime (EST anyways)

      • nakodari 6 years ago

        The issue started happening much sooner, around morning time in EST, but it got worse as the day progressed. It looks like the Amazon engineers woke up and realized something was wrong and they updated their status page 5-6 hours late. But I agree, AWS is super reliable, it's unfortunate that this problem occurred and I hope it's resolved soon.

wldlyinaccurate 6 years ago

From https://status.aws.amazon.com/

Intermittent DNS Resolution Errors

We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.

MastaX1 6 years ago

Seeing major issues here as well. All my modules were not working for the majority of the day and all images were not displaying. This is no doubt a Ddoss attack, I just don't see any other reason that the entire east coast AWS DNS servers would just stop responding all of a sudden. If the issue is not fixed by tomorrow our IT guy is gonna have a damn field day implementing all the temporary mitigation's.

faCeti0us 6 years ago

Intermittent DNS Resolution Errors We are investigating reports of occasional DNS resolution errors with Route 53 and our external DNS providers. We are actively working towards resolution.

https://status.aws.amazon.com/

  • Bob312371OP 6 years ago

    Why does the status page say resolved when S3 is still down for parts of US?

raghava 6 years ago

status check: Tue Oct 22 22:50:22 UTC 2019

DNS resolution to AWS S3 and RDS is still buggy on other providers (DigitalOcean, etc as well)

https://status.digitalocean.com/incidents/1z3kmlvz69v6

https://developercommunity.visualstudio.com/content/problem/...

joshuapinter 6 years ago

Having DNS issues with s3-us-west-2 as well.

DevX101 6 years ago

Having issues here in US East as well.

champbronc2 6 years ago

Seeing the same thing

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection