Last month our Tech & Society book club here in Bangkok read “Superintelligence: Paths, Dangers, Strategies” by Nick Bostrom. Book clubs are great by the way, highly recommend.
The first version of ChatGPT was released in 2022. Bostrom published Superintelligence in 2014. That’s 8 years before. This book is the bible the people owning the AI labs are preaching from, written before AI as we know it even existed.
Until a few weeks ago I didn’t know about this book. But when AI researchers talk about fast take-off scenarios, alignment or the problem of control, they are referencing Bostrom’s work.
The book was a dry and highly technical read. It goes deep into detail and has a huge number of references. But being this academic is what makes it so powerful: It creates authority. Superintelligence doesn’t invent new ideas. It legitimizes them by providing a solid foundation.
And if anything, this book showed me how powerful books and ideas can be.
Superintelligence (SI) is an intelligence that is vastly more intelligent than humans in every aspect, think at least 1000x. It doesn’t have to be artificial: it could also be biological, or something entirely alien.
For AI to become SI it first has to be AGI, Artificial General Intelligence, meaning in every aspect at least as intelligent as humans.
Bostrom speaks in facts and nuance. But at the same time he has a clear message for the reader: Humans creating a god-like Superintelligence is very plausible and needs to be taken seriously. And the dangers that come with it are very real problems without clear solutions.
When the book was released 12 years ago this still felt like science fiction. Now, with the speed of AI development, it suddenly feels much closer to reality.
It seems counter-intuitive that the same figures evangelizing for AI are also preaching the warnings Bostrom outlined in his book. If you believe SI is around the corner and the risks are real, why keep racing towards it? Or if you are not worried about SI and want to push AI, why talk about these dangers?
The answer you hear is that SI will happen no matter what. As technology advances, building SI becomes easier and easier, and sooner or later someone is going to do it. Do you want that someone to be an evil actor, or do you want it to be yourself?
But this is not a law of nature. It’s a choice dressed up as fate. If everyone in the race can say “I had no choice, someone else would have done it anyway,” then a collective choice gets made without anyone ever owning it.
You cannot warn about a danger, ask for regulation, keep building the very thing you warn about, and claim you have no choice.
On an individual level you always have a choice. You can stop building. You can focus entirely on safety, regulations and activism. If you think we should stop building, then you have the choice to stop building and walk away. And people did that. It is possible and it makes a difference.
What does the story do for the people telling it?
If you are the creator of a god, what does that make you? By definition the prospect of god-like superpowers is attractive to anyone looking for power.
And even if you don’t believe SI is coming soon, AI labs still profit from preaching its danger. There is no question that AI in its current state already gives enormous power to whoever controls it. To hold this much power, you need to justify to people why they should let you have it. What better reason than to save humanity from its biggest existential threat?
It justifies you wielding this much power, and it justifies any means necessary. Any sacrifice pales in comparison to the whole of humanity going extinct. It is also a convenient excuse to not address any other problems: Once SI is reality, everything else will be trivial to solve.
The key is to get people to believe you. The threat must be real enough that you can come in as the savior protecting humanity from it. That’s why Bostrom’s work is so powerful: it gives you authority when you say Superintelligence is real and inevitable.
I’m not saying everyone telling this story is cynical. Many mean every word. But beliefs and incentives can line up conveniently. And this story helps legitimize the absurd levels of power we are granting AI companies right now.
The story of a singleton Superintelligence with god-like power turning into our doom or our savior is really powerful. It feeds right into our human need for belief and meaning. And the story is told by powerful people and backed by scientific arguments. But it’s still a story. It’s not reality, and it’s not the only possible story for how reality might play out.
There is no denying that AI is extremely powerful. That doesn’t mean it has to turn into a god. It’s technology created by humans. And AI is not the only human creation posing existential risk: atomic weapons, viruses, hacker attacks. We have to treat AI as the lethal technology it is, and regulate it the way we regulate other dangerous technologies.
And if there is even a small chance of Superintelligence becoming reality in our lifetime, why would we assume humans have any chance of controlling a power potentially million times more intelligent than us? That is not an argument for a race towards it. It’s an argument for slowing down, for guardrails, for regulation like our lives depend on it. Because they do.
We hold the power to create our own story. Each of us can take steps towards the future we want to see. What does choice look like in practice? Citizens and lawmakers demanding real, independent regulation rather than rules written by the labs. Researchers and engineers who refuse to build what they say they fear. And all of us refusing to accept the framing that there is no alternative.
We don’t have to buy into the story we are told about Superintelligence. Each of us has the power to write our own story.
Frank Herbert said it best in Dune in 1965:
“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”