Fastly Outage

Well, with SO, at least you can search on Google and view the version cached by Google just fine.

With Reddit however, these days almost all comments are locked behind “view entire discussion” or “continue this thread”. In fact, just now I searched for something for which the most relevant discussion was on Reddit; Reddit was down so I opened the cached version, and was literally greeted by five “continue this thread”s and nothing else. What a joke.

Stack Overflow is down, can someone tell me how to declare a static multidimensional array in C++?

Haha! That sounds highly plausible!

Haha! That explains why the internet was down for a while!

Oh man, how do we keep a pocket copy of SO? All of our jobs depend on it.

While we're here.. I am a bit surprised to see how many sites use Fastly. As a dev I've always been happy with Cloudflare.

New error now, hopefully fix in progress.

Fastly error: unknown domain: www.fastly.com.

Details: cache-syd10161-SYD

Might not be the case anymore, but a few years back, Hackernews was just running on a single server.

I am already here

We use Fastly (and our site is down too) but I asked them about this a couple of years ago. It is deliberate. They said it was so they can tell if it is their Varnish service or the customer's Varnish service that is down

Fastly modified the Varnish error to ensure that it is known if the error is returned by Fastly's Varnish or by the origin's Varnish should the customer run their own Varnish on the origin.

Looks like an error (typo) with Varnish .. it's the same on multiple sites.

Maybe a good way to work out which versions are being used.

Mediation is a real word though?

This seems like it's intentional given the context.

I don't see the typo? definitely a fastly problem though..

Fastly error: unknown domain: numpy.org.

Details: cache-pdk17841-PDK

I assumed it was an intentional pun.

Down in SA

Down in UK

Does the 503 page load fast(ly) or slowly?

Cloudfront, by Amazon's own admission, specialises in high bandwidth delivery (ie huge videos). Fastly has consistently better performance as a small object cache, which makes it the choice for web assets

https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...

Fastly gives them the edge performance they need without having to build it themselves. They have been a customer for a while I think.

Yeah, this is what makes me feel this is more an AWS thing

I imagine they use a few different CDNs for things like this.

That doesn't appear to be the case does it? Amazon sites are all working fine, at least for me

Must be more than fastly. Heroku is also down.

GitHub Pages appears to be down too, taking an awful lot of sites offline

And this is (one reason) why using images instead of actual emojis is such a stupid idea. Why, Twitter, WHY?

The mirrors still work though, and cabal will just fall back to those

Not that the BBC are gloating that they're still up

Still a bit vague: https://twitter.com/fastly/status/1402221348659814411

I will never understand the meaning of putting CDN behind CDN.

My self hosted bitwarden is fine, as are all my self hosted sites.

and yet you're able to leave this comment.

if by "everything" you mean one thing, and by "centralize" you mean not centralized, then sure.

Why no other sites bypass the CDN and go directly?

If you have absolutely vanilla CDN requirements, you can run multiple CDNs and fail-over or load balance between them using DNS.

Quite a few Fastly customers have more than vanilla requirements though, and may have a lot of business logic performed within the CDN itself. That Fastly is "just Varnish" and you can perform powerful traffic manipulation is one of it's main selling points.

Use two CDNs and DNS providers for redundancy. Gets expensive, but at scale, probably doesn't make a huge difference. More complexity for the site operators to manage, however.

That's the problem with these black-box cloud offerings, that you can never know what will work (or not) and from where. You get semi-random, pseudo-localized outages that are not accounted for in all the 9's of availability.

With a standard TCP/UDP session, it mostly just works or doesn't and you can get a proper traceroute to know what's up. With these fancy CDNs, there's a whole new can of worms to deal with and from a client's perspective you have no clue what's happening because it's all taking place in their private network space where we have no "looking glass".

Fuck the cloud, i want real Internet.

Same here in central Poland (Łódź area), no problem with any of linked websites.

edit: My whole Twitter timeline is full of posts saying "Twitter outage? what outage?". Same on Reddit and Twitch chat, feels like for a short time I was invited into some exclusive circle lmao. StackOverflow and other StackExchange sites also work so I can look stuff up for you.

Interesting. Here in the Netherlands they don't.

Germany here (n=1), everything works except reddit and ft.com

depends where in France, most people i know here are affected as well

Not from Paris, France.

Not for me

That explains the spotty container build failures over the last half hour. Good thing I decided to procrastinate instead of debugging the issue!

Fastly largely runs on Varnish, it seems: https://www.fastly.com/blog/benefits-using-varnish

>At the core of Fastly is Varnish, an open source web accelerator that’s designed for high-performance content delivery. Varnish is the key to being able to accelerate dynamic content, APIs, and logic at the edge.

I think Fastly is the one having problems (they happen to use varnish but I haven't seen anything which says varnish is the root cause) - so all sites using it are down.

Why?

It's OK though, because large swathes of this discussion seem to have turned HN into reddit, at least temporarily. Normal service will no doubt resume in due course.

Edit: I didn’t mean anything negative here! Just slightly shocked that as the UK is opening up under 30 vaccinations, the US is struggling to find any more willing takers. It’s really probably a sign that there’s fewer anti-vaxxers in the UK more than anything. And that universal healthcare is more efficient at distribution than an inherently for profit system. I don’t know, but I just didn’t realize it was so different in the UK

Yes, because they're all just repeats of this one.

I hear that you're suggesting that those involved shouldnt feel bad because its a systemic / just a job / etc. But the reality is that incidents like this can be very traumatic for those involved and thats not something they can control. If it was that simple to manage, depression and anxiety would not be a thing.

Think its best to show a large amount of support and empathy for the individuals having a really bad day today, and how awful they may feel. Some will probably end up reading this thread (I know I would).

And of course, still hold Fastly the business accountable for their response (but objectively, once we understand what the root cause was, and the long term solution).

I think the government websites being down (UK ones for example) are the bigger issue. Reddit/Stackoverflow etc being down isn't that big of a deal imo.

I'm confused, why isn't being fired something to lose sleep over in your eyes?

I get that you're implying that the job itself is not worth that much concern, but it seems you're ignoring that jobs bring in income, pay your mortgage, etc.

If i lost my job tomorrow i'd be terrified.

Everyone's got responsibilities and aspirations. To be fair, I was thinking more of the jobbing engineer who's going to face anxiety about losing their job over this, but it extends to all levels. Having a fat bank balance helps get through periods without employment, but it's not just about money. There's anxiety, shame, embarrassment, the whole gamut. Going through a major incident at work is a shitty experience.

He's talking about Fastly themselves, not their customers

Just wait till they figure out how to make money off it.

The point isn't to dance around the incident, but to not blame people. You can blame systems, design, engineering culture, processes, but don't blame people. Even if someone accidentally pressed the 'destroy prod' button, that's not the fault of that person, it's the fault of that button existing and being accessible in the first place.

I have no empathy for Fastly-the-company. I hate the fact that the Internet is centralized around CDNs. I wish this idea of 'but we _must_ run a CDN for our 1QPM blog!' would die in a fire. But I can still empathize with the Fastly engineers handling this shitstorm right now.

Blame culture isn't the way forward here.

Do a post-mortem, work out root causes, work as a unit to ensure this doesn't happen again.

Obviously if there are levels of gross negligence or misconduct discovered during post-mortem, that will need to be dealt with accordingly, but coming into this with an attitude of "we must find someone to blame and incur repercussions" isn't healthy at all.

We are humans - don't forget that.

edit: forgot some words.

The problem is a blame culture ensures the near-misses are never reported. Air safety discovered this many years back - a no-blame culture ensures anything safety-related can be reported without fear of repercussions. This allows you to discover near misses due to human error and ensure that the overall system gains resilience over time. If you blame people for mistakes, they cover the non-obvious ones up, and so you cannot protect against similar ones in future, so your reliability/safety ends up much lower in the long run. It's all about evolving a system that is resilient to human error - we will make mistakes, but the system overall should catch them before they become catastrophies. In air travel now, the remaining errors almost never have a single simple cause, except in airlines/countries that don't have an effective safety reporting culture.

I recommend reading about "blameless postmortems" [1]. Our natural tendency is to look for who is responsible for an incident and point the finger of blame. Over time this leads to a cover-your-ass culture, whether you like it or not. Therefore such a tendency needs to be actively fought against to keep the focus on quality engineering and not politics.

"An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization."

[1] https://sre.google/sre-book/postmortem-culture/

I'm sure you've never made a mistake.

The best way (in a team), to tackle mistakes, is to ensure the process in place corrects these mistakes. The only way to do that, is a post-mortem/learning from the mistake. If you blame it on some engineer who did it, that guy will eventually be replaced by some other guy, who may make the same mistake.

We need to learn from our, and other mistakes, or else we keep repeating them. Nothing "old fashioned" about that.

And we, especially companies, typically only learn if there is something at stake. Stock-price, a job, customers, liability etc.

(Call me old fashioned, but what I learned from it, having no stake in the game, is we are truly demolishing the resilient, decentralised nature of the internet; or already have done so)

I don't agree about the blame, but I do also find the empathy cringeworthy. Something's broken; someone's job is to fix it; they'll fix it; it will work again. /shrug/

Post-mortems make far more interesting submissions IMO, but I suppose people up-vote 'yes down for me too'.

the attitude that "people need to be blamed" will never improve reliability in the long run. people come and go; systems and processes endure. blaming people is the best way to avoid making durable improvements to systems and processes.

I hear you, but I just want to point out that this rarely happens anywhere else. It's great if tech (and people in general) hold themselves to progressively higher standards than what is out there already, but I don't think tech needs to be that much better, I'd settle for just doing a good honest retro (without throwing anyone under the bus, and without covering their asses)

A good leader will take the hit (and the repercussions) for their underlings, compensate customers where compensation can make it better (and offer to make it easy to use fallbacks if this happens again) -- and internally fix the problem so it can't happen again, without throwing anyone to the dogs.

> People need to be blamed, and responsibility for actions taken (without covering asses)

What i think this syntactically invalid sentence is trying to say is:

People need to be blamed, and held responsible for actions taken.

Why do people need to be blamed? Why do we need to make someone the scapegoat? What does being held responsible look like?

Let say we find some sacrificial engineer to pin this on:

* does the downtime magically disappear?

* does the engineer suffering (say losing his job or whatever) make your downtime meaningful? You'll recoup your revenue somehow from it?

* does the fact that there's a scapegoat mean that everyone else at fastly is perfect and it's ok to keep using them?

Scapegoating in those situations happens more often than not. In an operations team all problems are systemic - having to do with decision makers throughout the process, sometimes acting on perverse incentives set up by others. Blame then gets diluted but still tends to fall upon the organization responsible rather than an individual, which is where it should be. Gross negligence is not so cut and dry.

"Call me old-fashioned but..." is a dog-whistle harking back to "better days" that never existed.

Emapthy and responsiblity are not mutually exclusive.

> People need to be blamed, and responsibility for actions taken (without covering asses)

This. When people talk about "HugOps", "empathy" and all that when a worldwide incident affecting a huge amount of time critical customers (e.g. trading, hft, cargo, food delivery, etc.) is happening for an hour, it has catastrophic consequences.

I hope the engineers also understand the other side and why we are paying huge sums of cash for their service.

I suspect you'll have trouble convincing a forum of primarily engineers that a high frequency trader is more worthy of sympathy than an engineer. They're both pretty privileged jobs and HFT is not known for having tons of benefits to society

And ignore the pre-agreed SLA targets and compensation for not meeting those targets that's in the contract they signed right? If you're going to say you're losing $X/minute of downtime, then either deal with it, architect around it, or negotiate the necessary SLA and compensation.

> this worldwide outage caused by them just goes to show you that Fastly does not actually have a failover system in place.

I don’t know Fastly at all, but in my experience there’s no such thing as a foolproof failover system that covers all possible scenarios.

> Not my problem. Fastly should work as intended.

What's your SLA with them?

Just assuming things will always work because the marketing copy said so is recipe for disaster. It's hoping that things never go wrong, and when they inevitably do, being caught pants down.

Everything fails sometimes. You must know how much your SaaS provider contractually promises, ensure that any SLA breach is something financially acceptable for you, and ensure that you can handle failure time within SLA.

yeah, the internet is working perfectly. if you want to view 503 errors.

Thank god network people haven't drunk the centralization kool-aid.

Interestingly, server side rendered pages worked well during the outage. Most of the issues were caused by sites that are relying too much on Javascript.

And only those websites on some networks. If I connect my phone to my cell network instead of wifi, the problem sites work for me.

"I'm old enough to remember when the Internet wasn't a group of five websites, each consisting of screenshots of text from the other four."

-- https://twitter.com/tveastman/status/1069674780826071040

:-(

Most people don't need any of them to continue with their life though!

BGP (the protocol underpinning the internet) is built entirely for avoiding outages of any size.

Even email has a method baked into to the protocol for handling failure.

Fallbacks are good, baking in resiliency is better.

Stuff is down across the web, but the most it says is “degraded performance” and in my area it’s all green even though the sites are still down.

I didn’t see any error whatsoever on their homepage, while now I see “Global CDN Disruption” on their status page.

Bit pedantic, but it's PyPI that Fastly gives services to, not pip (and PyPI that's down, not pip). The two are only loosely related – pip is a piece of software.

Gov.UK is back up too. They have a mandate from government to be able to provide emergency communications so I expect they did have a backup and have managed to switch over, but just took 30 mins to do so.

Gov.UK is supposed to be a bit like BBC 1 or Radio 1 – in a national emergency they can be taken over to disseminate critical information, like if there was a nuclear attack launched on the UK.

The mirrors still work though, and cabal will just fall back to those

For sites of any complexity with any dynamic content having CDN redundancy is akin to being multi-cloud -- it is not worth the effort.

A lot of dynamic sites use Fastly for its programmatic edge control and a near immediate ( ~1s-4s, typically around 2 ) global cache invalidation for any tagged objects with a single call to the tag. That feature alone simplifies backend logic significantly. To make this feature portable to CDNs that do not support it and provide only regular cache invalidation requires a complicated workflow setup which significantly increases the cache bust time, which in turn removes all the advantages of the treat dynamic content as static and cache bust on write approach.

Lowkey status reports are the norm now :)

"some users may experience degraded service" => site completely down for all locations

Yeah, I also wrote a bot that chooses to create a status incident with the lowest key neutral message when it detects continued healthcheck fails (outside of maintenance) that steps in if an operator hasn't already created an incident. Maybe they're too busy fixing.

Yes, I also thought the header was hilarious:

> CDN Performance Impact

Im experiencing the outage here in brissy its not looking good

They've updated it to

>North America (Ashburn (BWI), Ashburn (DCA), Ashburn (IAD)), Europe (Amsterdam (AMS)), and Asia/Pacific (Hong Kong (HKG), Tokyo (TYO), Singapore (QPG)).

Seems like they are still taking stock of exactly what is broken.

It has now been updated to a pretty sizable list.

edit: And now it looks like it includes every location.

They used to use AWS CloudFront and switched to Fastly, someone shared this in another comment:

[https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...: CDN Fastly Wins Content Delivery Business For Amazon.com and IMDB Websites)

Quoting:

> "But with small object delivery, like images loading fast on Amazon’s home page, it’s the opposite. Customers will pay for a better level of performance and in this case, Fastly clearly outperformed Amazon’s own CDN CloudFront. This isn’t too surprising since CloudFront’s strength isn’t web performance, or even live streaming, but rather on-demand delivery of video and downloads."

Amazon (like a lot of others) use several CDNs for redundancy. You can see from dig that it resolves to combinations of cloudfront, akamai, and (presumably, based on your reported experience) fastly.

  dig +short www.amazon.com
  tp.47cf2c8c9-frontier.amazon.com.
  d3ag4hukkh62yn.cloudfront.net.
  65.8.70.16

  dig +short www.amazon.co.uk
  tp.bfbdc3ca1-frontier.amazon.co.uk.
  dmv2chczz9u6u.cloudfront.net.
  13.224.0.89

  dig +short www.amazon.in
  tp.c95e7e602-frontier.amazon.in.
  d1elgm1ww0d6wo.cloudfront.net.
  13.224.9.30

  dig +short www.amazon.co.jp
  tp.4d5ad1d2b-frontier.amazon.co.jp.
  www.amazon.co.jp.edgekey.net.
  e15312.a.akamaiedge.net.
  104.71.134.162

You're right, I should've said *partially* back. At least the CSSs now load, but a few products images are still gone. However it was completely broken here before (literally loading just the main HTML).

Unless you're browsing reddit without logging in, you can just set the old reddit theme from your account settings so you don't need to use the old. prefix :)

gasp you're absolutely right...!

Seems to be mixed for me, BBC News and Sport works but stuff like Weather, iPlayer (video streaming) and Sounds (audio streaming) have died. I guess the BBC is big enough that different bits of the site run off different solutions (perhaps news and sport are still in spirit running off "news.bbc.co.uk" instead of the main servers?).

Not here (although won't be long)

dig bbc.co.uk

  bbc.co.uk.  193 IN A 151.101.64.81
  bbc.co.uk.  193 IN A 151.101.128.81
  bbc.co.uk.  193 IN A 151.101.192.81
  bbc.co.uk.  193 IN A 151.101.0.81

it's down

Would be interesting to know what these fail over patterns are. As DNS takes a while to propagate, I thought DNS records already indicate fail over addresses.

> In HTML/CSS you should be able to specify a fallback source if the first returns a non-200.

Except if the HTML/CSS is hosted on that CDN?

DNS didn't fail, and there's nothing you can do in HTML/CS/JS if your CDN fails to serve those things

Content-centric networking had been a central research topic for many years. And many potentially useful systems have been proposed and implemented.

At some point some of them will start to become popular.

Web Browsers should probably retry a different server in DNS if they get a 503 - but they don't.

I don't think you understand how ecommerce works.

A very significant amount of people won't go back. It's why the most effective marketing campaign by far is retargeting those people to convince them to come back. Unfortunately that's not possible in this case since you can't track the users as the site is unusable.

Makes me wonder how the engineers will fix this if they can’t visit Stack Overflow :)

Believe it or not, but there are developers out there that read the docs.

As per report above - most (or all?) of Asia/Pac servers are down.

This incident affects: North America (Ashburn (BWI), Ashburn (DCA), Ashburn (IAD)), Europe (Amsterdam (AMS)), and Asia/Pacific (Hong Kong (HKG), Tokyo (TYO), Singapore (QPG)).

Ah, I meant more sites like ABC, 9NOW, SBS, AFL, Foxtel etc rather than accessing US sites from AU.

I’d love to see a breakdown of what single point of failure causes these worldwide network outages. They even brag about redundancy in their marketing materials. I hope we see a post mortem on this

> Agree, but currently, ipfs would serve as a fallback, since it's about files.

Isn't a CDN fundamentally all about files too?

> Decentralized/distributed generally has slower network performance. Unless most nodes are high performance, I guess?

There is definitely more work to do here before this is really useful, but it's well within the realm of things that IPFS should be able to do at reasonable performance for production sites in future. Good performance still requires a serious CDN node network similar to traditional CDNs today (to seed your content for day to day use) but with IPFS if that CDN goes down then existing users on your site can _also_ serve the site to other nearby users directly, or other CDNs can serve your site too, etc etc. Your DNS wouldn't be linked to any specific CDN in any way, just to the hash of the content itself, so anybody could serve it.

> Decentralizing the internet works if it financially makes sense for platforms to build such tools.

There's a platform company called Fleek who already do this today: https://fleek.co/hosting/ (no affiliation, and I've never even used the product, just looks cool). Seems to be designed as a Netlify competitor: push code with git and it builds it into static content and then deploys to IPFS.

The benefits don't exist today of course, because no browsers natively support IPFS, so most users can only access the content via an IPFS gateway, which means you're back to fully centralized server infrastructure again... If we can get IPFS support into browsers though then fully decentralized CDN infrastructure for the web is totally possible.

I mean, yes, absolutely, and that works to start with, but I'm willing to bet the overall uptime and performance of a raspberry pi in your living room is quite a bit worse that Fastly's :-).

[deleted]

Various bits of GitHub on the Web (committing edits, editing releases) were broken for the same reason. Failure modes of JS-heavy GUIs are interesting.

Looks like it is. If you're still able to see much of the UI, don't force-reload the page as it'll invalidate the CSS in the cache.

I did that moments ago, and I regret it.

Thanks. That makes sense.

Wouldn’t you build in a failsafe that bypasses Fastly and sends traffic to your own servers in the case of this kind of outage? Or outages are so rare that it’s not worth the trouble?

Amazon doesn’t enforce dogfooding, their retail site is its own stack and has been only recently been migrating to EC2s

[deleted]

You might be right. Here is another update from fastly:

The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return.

Let's see

We've got Cloudflare sitting in front of our Firebase/GCP instance (which I've just found out is Fastly-cached :/). Getting 503s at the origin but we're up on our URL with an always online notice thanks to CF. Double dip isn't all that bad.

It's always DNS...

https://xkcd.com/2347/

Maybe this one, titled "The Cloud". https://xkcd.com/908/

We have no way to know. https://xkcd.com/908/

Might be xkcd.com/503.

You have put your finger on it. I AM bitter about it. It used to be really cool, and really nice to use, before the Taylor/Pao dustup, and the redesign.