We turned off Pub/Sub and nobody noticed

I like the article, but I think there is a clear misconception of what a SPOF is, and I've noticed this happening across many engineers and engineering teams lately.

A system, like Pub/Sub, being a critical dependency or hard dependency does not make it a SPOF. Otherwise any system is a SPOF, creating absurd architectures that end up with more complexity and downtime than the original one. For example, I assume incident.io are yet to see if their added complexity of multiple event streams does not cause worse availability issues than what they had, by introducing complexity like problems with data consistency.

If the the underlying "system" components are not redundant and highly available, then you have a SPOF. eg. A single machine is what provides Pub/Sub.

In this case, what you have is a hard-dependancy with an avaialbility lower than what you want to achieve, but there are always shared hard-dependancies that they will have. There is no way around this. What we should aim for is that any system that is a critical dependancy or a hard dependancy with such a big blast radious, to be redundant and robust in itself. Not just the system, but the practices around it.

Disclaimer: I work at the incident company

This is one of those projects where the end goal is to build the scariest button possible. You cannot complete the project until you’ve pressed that button. You do all the work to build the button, then you have to press it.

Never not scary, never not eventually fun, but type 2 fun for sure.

NATS is such a cool piece of technology that deserves more hype than it gets