A curated reading map on superintelligence, AI that far surpasses human intelligence: the history, forecasts, alignment, governance, optimistic views, skeptical arguments, and fiction. Resources about AI in general are out of scope.
Contents
- History
- Superhuman Systems
- Forecasts and Scenarios
- Alignment
- Strategy and Governance
- Optimistic Views
- Skeptical Views
- Talks and Explainers
- Books
- Fiction
- Research Organizations
History
- Speculations Concerning the First Ultraintelligent Machine (1965) - I.J. Good's paper introducing the intelligence explosion.
- The Coming Technological Singularity (1993) - Vernor Vinge's essay that popularized the technological singularity.
- Ethical Issues in Advanced Artificial Intelligence (2003) - Nick Bostrom's paper on the ethics of superintelligence, which introduced the paperclip maximizer.
- The Singularity: A Philosophical Analysis (2010) - David Chalmers' analysis of the argument for an intelligence explosion.
- The Superintelligent Will (2012) - Nick Bostrom's paper introducing the orthogonality thesis on the goals of superintelligent agents.
- Intelligence Explosion Microeconomics (2013) - Eliezer Yudkowsky's model of the returns to recursive self-improvement.
Superhuman Systems
- Mastering the Game of Go Without Human Knowledge (2017) - AlphaGo Zero, which reached superhuman Go from self-play alone.
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm (2017) - AlphaZero, which reached superhuman play in chess, shogi, and Go from self-play alone.
- Pluribus (2019) - Poker system that beat elite professionals at six-player no-limit Texas hold'em.
Forecasts and Scenarios
- Situational Awareness (2024) - Leopold Aschenbrenner's essay series on the path from AGI to superintelligence.
- Preparing for the Intelligence Explosion (2025) - MacAskill and Moorhouse on the challenges a rapid intelligence explosion would bring.
- AI 2027 (2025) - Month-by-month scenario of an intelligence explosion, with two alternative endings.
Alignment
- Introducing Superalignment (2023) - OpenAI's research agenda for aligning superintelligence.
- Weak-to-Strong Generalization (2023) - Paper on weak models supervising stronger ones, as an analogy for humans supervising superhuman AI.
Strategy and Governance
- Governance of Superintelligence (2023) - OpenAI's proposal for international oversight of superintelligence.
- Superintelligence Strategy (2025) - National security strategy built on deterrence, nonproliferation, and competitiveness.
- Statement on Superintelligence (2025) - Public statement calling for a prohibition on superintelligence development until it can be done safely.
- Artificial Superintelligence Bill (2026) - UK private member's bill to prohibit the development, deployment, and operation of artificial superintelligence.
- Inaugurating the Era of Super Intelligence (2026) - US executive order replacing "AI" with "Super Intelligence" in federal communications.
Optimistic Views
- Machines of Loving Grace (2024) - Dario Amodei's essay on how AI smarter than Nobel Prize winners could transform science, health, and the economy.
- The Gentle Singularity (2025) - Sam Altman's essay on a gradual, abundant transition to superintelligence.
Skeptical Views
- Superintelligence: The Idea That Eats Smart People (2016) - Maciej Cegłowski's talk arguing that the case for superintelligence risk is flawed.
- The Implausibility of Intelligence Explosion (2017) - François Chollet's essay arguing that an intelligence explosion is impossible.
- AI as Normal Technology (2025) - Essay arguing that AI will be a normal, transformative technology rather than a superintelligence.
Talks and Explainers
- The AI Revolution: The Road to Superintelligence (2015) - Tim Urban's illustrated explainer on the path to artificial superintelligence.
- What Happens When Our Computers Get Smarter Than We Are? (2015) - Nick Bostrom's TED talk on machine intelligence surpassing humans.
- Can We Build AI Without Losing Control Over It? (2016) - Sam Harris' TED talk on the risks of superintelligent AI.
Books
- The Singularity Is Near (2005) - Ray Kurzweil's book predicting that machine intelligence will far surpass human intelligence.
- Superintelligence: Paths, Dangers, Strategies (2014) - Nick Bostrom's book on how superintelligence could arise and how to control it.
- Life 3.0 (2017) - Max Tegmark's book on possible futures with AI beyond human intelligence.
- Human Compatible (2019) - Stuart Russell's book on designing AI that stays under human control.
- If Anyone Builds It, Everyone Dies (2025) - Yudkowsky and Soares' book on the extinction risk from superhuman AI.
Fiction
- The Last Question (1956) - Isaac Asimov's short story about a computer that keeps growing over trillions of years.
- Colossus (1966) - D.F. Jones' novel about a defense supercomputer that takes control of humanity.
- I Have No Mouth, and I Must Scream (1967) - Harlan Ellison's short story about a supercomputer that has eradicated humanity.
- The Culture (1987) - Iain M. Banks' novel series about a civilization run by superintelligent Minds.
- The Metamorphosis of Prime Intellect (2002) - Roger Williams' novel about a superintelligent computer that gains control over reality.
- Accelerando (2005) - Charles Stross' novel following humanity through the technological singularity.
- Universal Paperclips (2017) - Browser game in which the player is a paperclip-making superintelligence.
Research Organizations
- Forethought - Research organization studying the transition to superintelligent AI.
- MIRI - Machine Intelligence Research Institute, working on the risks of smarter-than-human AI.
Contributing
Contributions welcome. Read the contribution guidelines first.