Settings

Theme

Tell HN: Both recent GCP outages caused by fiber optic maintenance

20 points by fastest963 · 4 comments · 1 min read


Both of GCP's recent outages were caused by "fiber optic maintenance". On August 20th when us-west1 went down they said [1]:

> The disruption originated during scheduled fiber optic maintenance, which unexpectedly compromised network capacity between data centers within the us-west1 region.

Then on September 1st when us-central1-b (and collateral affects in other zones) went down they said [2]:

> The immediate technical trigger for this event was the inadvertent physical disconnection of network fiber-optic cables during a routine hardware maintenance procedure.

Not sure if they're related but apparently procedures we're significantly changed after the first incident to prevent the second incident.

[1] https://status.cloud.google.com/incidents/utF3FMFdQfwBzJcGG6vf

[2] https://status.cloud.google.com/incidents/J5ia5t9p3g9Q5Wi7r8Ev

1 thread
mjuarez

> Not sure if they're related but apparently procedures we're significantly changed after the first incident to prevent the second incident.

Which obviously didn't work. Or worse, made the second incident more likely.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection