Underwriting Superintelligence

Underwriting
Superintelligence

Rune Kvist, Rajiv Dattani, Brandon Wang

July 15, 2025

Insurance Unlocks Secure AI Progress

We’re navigating a tightrope as Superintelligence nears. If the West slows down unilaterally, China could dominate the 21st century. If we accelerate recklessly, accidents will halt progress, as with nuclear power.

Insurance, standards, and audits together create skin in the game for quantifying, communicating, and reducing AI risks so we can balance this tightrope. We call this the “Incentive Flywheel.”

Benjamin Franklin first discovered the Incentive Flywheel, when fires threatened Philadelphia’s growth. He gathered neighbors and founded America's first fire insurance company. They created volunteer fire departments and established the first building safety standards.

Since then, this Flywheel has been at the heart of balancing progress and security for new technology waves like electricity and the automobile.

But the Incentive Flywheel won’t appear fast enough on its own for AI: we need to jumpstart it. This essay outlines 25 actions entrepreneurs and policymakers must take by 2030 across agents, foundation models, and data centers.

Markets are a uniquely Western solution to risks. The Incentive Flywheel adapts faster than regulation, accelerates rather than slowing down progress, and has more teeth than voluntary commitments.

Benjamin Franklin and the Incentive Flywheel

Houses in Philadelphia in the 1700s had a bad habit of burning down. Made of wood and packed closely together, fire caught easily and spread quickly, killing many. Homeowners could not assess their own fire risk. And did not bear the full cost of their negligence. Ad-hoc volunteer responses failed. A single uncontained fire would often destroy entire city blocks.

As the population of Philadelphia grew tenfold in the 1700s, residents were building houses faster than the systems meant to contain them.

Franklin’s solution shows up again and again through history and offers a blueprint for how markets can speed up secure adoption of new technology.

AI capabilities are advancing at breakneck speed, from preschool-level intelligence in 2020 to undergraduate-level in 2024. Many researchers expect superhuman AI as early as 2028, potentially creating “a country of geniuses in a datacenter.” AI could help us develop cancer treatments that can reduce mortality by 95%; on the other hand, AI hackers can already find holes in our cyber security defenses, and the same capabilities could soon enable terrorists to create synthetic bioweapons. Epochs are increasingly measured in months or weeks.

To capture AI’s benefits and increase America’s competitive lead, we must move forward with haste. The civilization with the most powerful AI will be economically, culturally and militarily superior. In the last two years, the US lead over China in AI capabilities has shrunk from 2 years to less than 8 months. Accelerationists are correct that most regulation will slow AI progress down, risking our already tenuous lead.

On the other hand, cautious voices are also correct that we may race right into the wall, unless we steer with care. Racing for market share is fine when the risks are small (buggy software rarely kills people). But we don’t want medical device manufacturers or nuclear power plant operators to move fast and break things. AI will quickly get baked into critical infrastructure and could enable dangerous misuse. The territory is sufficiently uncharted that any incentives for cutting corners should make us nervous.

Speed of AI progress is often presented — incorrectly — as being in tension with AI security.

Progress requires security. An accident could cause significant damage, and threaten America’s lead in AI. Nuclear's promise of abundant energy died for a generation after accidents like Three Mile Island and Chernobyl accelerated public backlash and regulatory scrutiny. The same will be true if AI causes major harm — courts and voters will shut AI progress down.

Security powers progress. ChatGPT was created using an AI alignment technique called RLHF that made systems more steerable — and thus more useful. Steerable, reliable AI systems are simply more valuable.

More secure than voluntary commitments from AI companies: the rapid pace of AI progress and associated catastrophic risks mean that AI companies’ voluntary commitments will not inherently create security. The flywheel will align incentives and create accountability.

Faster than regulation: major pieces of regulation, created by bureaucrats without technical expertise, move at glacial pace. Patchwork regulation across 1,000+ state bills will create a burden on AI developers that slows adoption and investment. The flywheel will move faster and create the confidence necessary for adoption.

Figure 1: The Incentive Flywheel of Market-based Governance

INCENTIVE FLYWHEEL

The market mechanics are already taking shape:

Insurance pays when things fail, motivating insurers to demand proof that risk is well-managed and fund research into lowering risks even further. This is more than risk transfer: insurers often require best practices to be followed before offering insurance. This is happening today with customer service companies starting to buy insurance against AI hallucinations that cause financial losses, like the one suffered by Air Canada where a chatbot hallucinated the airlines refund policy. As insurers accurately assess risk through technical testing, coverage becomes both a credible safety signal and protection if AI fails. This enables faster adoption with confidence. Another example emerged in late 2023 when Microsoft announced a “Copilot Copyright Commitment” effectively insuring customers against copyright violations. Within months OpenAI, Anthropic, and Amazon had all implemented versions of this copyright insurance for code-generation models, while Adobe, Canva, and Getty offered similar for image generation. These promises have now become a competitive vector, with many enterprises requiring them of vendors, thereby forcing smaller vendors to purchase these policies from insurers. This incentivizes the development of safeguards and monitoring required by the insurer.

Audits assess risk independently to let insurers price that risk accurately and inform insurance eligibility. The best AI developers already test their products and share the results with customers for assurance. Independent evaluations give buyers and insurers confidence to adopt and accurately price risk. Again, insurers often run or fund audits because they need risk data. This incentivizes robust audits.

Standards boost communication in markets. They distill complex topics into easy-to-understand certificates and metrics. They clarify best practices and what it means to take ‘due care’ that informs liability and audits. For AI model developers, defining specific risks and thresholds at which system mitigations are required is becoming common practice. For example, as AI models get better at biology, we need to prepare for new biothreats. For data centers, initial work has defined threats and thresholds for Security-Levels, and further work is ongoing to build an auditable standard. Standards are often funded by insurers. This incentivizes the reduction of damages.

Once this flywheel is spinning, investing in security will enable AI companies to grow faster by enabling confident customer adoption. Standards and audits help enterprise risk teams distinguish hype from reality, just as bond ratings help (1) investors act with confidence and (2) governments and regulators oversee financial institutions.

Historical Blueprint: Fire & Car Safety

This is not a new model.

When electricity created new fire hazards around the turn of the 20th century, Chicago Fire Underwriters' Association and the Western Insurance Union funded Underwriters Laboratories (UL) to research risks, certify products, and develop safety standards. The lightbulbs and toasters in your house today are almost certainly UL certified and marked today.

When demand for cars increased after WWII, the insurance industry established the Insurance Institute for Highway Safety (IIHS) in 1959, nearly a decade before federal government action. IIHS ratings and premium discounts created direct incentives to adopt seatbelts and airbags before they became mandatory. Deaths per mile plummeted 80% while driving surged 200%.

This Flywheel reduced risks, letting entrepreneurs build governance capacity long before government intervention.

Skin in the game is the driving force at play. Financial markets rely on risk assessments, like Moody's assigned AAA bond ratings to toxic mortgage securities before 2008, because they were paid by issuers, not affected by losses. Insurance is therefore the necessary skin in the game: when insurers misprice risk, they go bankrupt.

Even within AI, this idea of using market-based approaches to manage AI development, or using insurance, standards, and audits is not new. The rest of this piece explores in more detail how to get the flywheel turning faster, including investments and decisions required by players in the ecosystem.

Agents, Foundation Models, and Data Centers

The Incentive Flywheel secures AI progress across all three critical layers of AI development:

Applications represent the majority of AI agent deployments in the real-world today. Enterprises must adopt AI agents to maintain competitiveness domestically and internationally.

Foundation model developers are racing to build superintelligence. They must build the confidence of their customers and stakeholders including the public to earn the right to continue investing in, and deploying these capabilities.

Data center developer infrastructure is critical for the application and model developers. They must build the confidence of their customers and stakeholders (including governments) to earn the right to scale investments to trillions of dollars and protect what could become the most valuable asset in the world.

Figure 2: Applying the Incentive Flywheel across AI development

Standards in particular are central to leading global AI development. Industry standards shape technological development, and they’re increasingly a central geopolitical battleground. China recently boosted its presence in international standards bodies; the US recently renamed a core AI group to Center for AI Standards and Innovation (CAISI). Private industry is central here: for example, China’s Huawei is now the largest provider of 5G equipment in the world. AI must be built on American infrastructure.

Faster Than Legislation

In the absence of market-based approaches, many people are looking to legislation and regulation to fill the gap. The result of this is a broad range of legislation is working its way through various states and countries. Over 1,000 state-level AI bills have been introduced in the U.S. in 2025 alone. The bills are mostly well-intentioned, and those who are cautious about AI are right about their core concern. However, disconnected regulation creates a burdensome patchwork of divergent compliance regimes that will hold up investment and adoption.

Crafting comprehensive laws like the EU AI Act takes longer than it took for AI capabilities to advance from preschool to undergraduate level intelligence. In the last two years, two factors have completely changed the regulatory premise: token costs have dropped by more than 99% while open-source alternatives have emerged.

Markets leading regulation is a more effective way to satisfy all parties. For most types of risk, insurers are incentivized to develop and quickly iterate on core safety measures. Those risks can then be codified into fewer, more simplified pieces of regulation once proven (e.g. mandating airbags). Market-based governance prices in risk changes in real-time and insurance rates adjust monthly based on new data, enabling markets to clear the fog.

Other markets show how even in scenarios with moral hazard (e.g. car accidents) or catastrophic risk (e.g. nuclear and terrorism) markets can carry the majority of the risk, supported with government mandates and backstops. For nuclear, the Price-Anderson Act of 1957 codified that the private sector cover losses up to $16 billion, with the government covering truly catastrophic excess costs. Three Mile Island's cleanup in 1979 was fully covered by private insurance. Tail risk can be difficult to price, but the fact remains: someone is always implicitly underwriting that risk, with or without insurance.

Other markets show how even in scenarios with moral hazard (e.g. car accidents) or catastrophic risk (e.g. nuclear and terrorism) markets can carry the majority of the risk, with government mandates and backstops supporting. For nuclear, the Price-Anderson Act of 1957 codified that the private sector cover losses up to $16 billion, with the government covering truly catastrophic excess costs. Three Mile Island's cleanup in 1979 was fully covered by private insurance. Tail risk can be difficult to price, but the fact remains: someone is always implicitly underwriting that risk, with or without insurance.

Only the government can deal with certain national security risks (e.g. ensuring international proliferation of standards, secure critical infrastructure, and national defense). In these areas governments should lead, partnering with the market to support the development of technologies and deployment as needed.

More Secure Than Voluntary Commitments

The accelerationist approach correctly identifies that markets excel at experimentation, learning, and adaptation. Capital chases promising ideas and bad products disappear over time (e.g. FTX crashed, while Coinbase thrives). However, the nascent AI markets suffer from market failures to prevent secure-by-default outcomes. Misaligned incentives and speed of progress mean companies do not face the consequences of cutting corners, while customers and investors lack the information to accurately assess security. There is a missing market to address these challenges:

Figure 3: Summary of why voluntary commitments are insufficient for secure AI progress

Companies that invest more in security testing get to market slower and are beaten by competition. This has led to a streak of AI models being deployed without sufficient testing leading to roll-backs and broken commitments.

The Flywheel Is Already Emerging

We must develop the mechanisms to secure AI progress faster than we have for any technology before. Car safety standards took more than 40 years to bring down the rate of death. Artificial superintelligence has significantly higher potential for both good and harm than cars, and it could be less than 4 years until it is upon us. However, there are three reasons to be optimistic: all players profit from faster adoption resulting from more security. History provides a blueprint we can learn from. And there are early signs of the flywheel emerging today.

Established insurers like Munich Re (Est. 1880) have teams dedicated to addressing generative AI risks. Cyber insurance companies like Coalition and Resilience (both valued at $1B+) have proven how to bundle insurance with deep technical expertise. Organizations like METR, Transluce, Haize Labs, and Virtue AI are pushing the technical evaluation frontiers. AI labs coalesce around “Frontier AI Safety Commitments" and share information and best practices through the Frontier Model Forum, while NIST has published AI Risk Management frameworks.

25 Immediate Actions to Accelerate the Flywheel

This emerging progress is insufficient. The scale and risk of this opportunity demands that we collectively invest $100+ billion over the next 5 years to create insurance policies, strengthen technical auditing capabilities, and develop industry standards. This investment will represent only a single digit percentage of spend on AI progress. By 2030, global AI data centers alone are projected to require $5 trillion of investment, while enterprise AI spend is forecast to reach $500 billion. Comparatively, cybersecurity budgets are ~6% of IT spend and airports spend ~20% of their budgets on security.

Below are 25 actions required in the coming years. Most can be led by private industry. The actions focus on what we will need in the coming years, but we can get started with much less. The place to start is insuring the near-term harms that already have clear liability, or where contractual indemnity can be established. For example in the case of agents: hallucinations, IP infringement, bias, harmful outputs. Insuring these risks with AI-specific insurance will incentivize data-collection across risk types, research into standards, and adherence to these best practices from developers. Insuring million dollar risks will pave the way to insuring the billion dollar risks.

Figure 4: 25 Immediate Actions to Accelerate the Flywheel

1. Insurance leaders to update policies to explicitly include or exclude AI risks from coverage, providing $500bn+ of AI coverage by 2030. Avoid a repeat of cyberinsurance, where insurers left customers unsure of whether cyber attacks were covered by their insurance policies and trillions of dollars of exposure remain without protection 2. Insurers to require that AI security best practices from standards are adopted to be eligible for insurance, like insurers did to drive adoption of early Multi Factor Authentication 3. Industry to fund data collection of AI incidents to support underwriting and standards similar to insurer-funded PERILS tracking natural catastrophe data

1. Insurance leaders to update policies to explicitly include or exclude AI risks from coverage, providing $500bn+ of AI coverage by 2030. Avoid a repeat of cyber-insurance, where insurers left customers unsure of whether cyber attacks were covered by their insurance policies and trillions of dollars of exposure remain without protection 2. Insurers to require that AI security best practices from standards are adopted to be eligible for insurance, like insurers did to drive adoption of early Multi-factor Authentication 3. Industry to fund data collection of AI incidents to support underwriting and standards similar to insurer-funded PERILS tracking natural catastrophe data

9. Industry convening to build $25bn+ coverage for foundation model developers, syndicated across the industry (akin to autonomous vehicles or aerospace), with data sharing and building risk modelling infrastructure 10. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and model developers confidence (akin to Price-Anderson Act for nuclear energy) 11. Clarification of legal liability to create clarity and accountability (akin to Price-Anderson Act) 12. Scale funding from $150m today to $500m+ annually for research into technical auditing of developers 13. Require adherence to a single industry standard

9. Industry convening to build $25bn+ coverage for foundation model developers, syndicated across the industry (akin to autonomous vehicles or aerospace), with data sharing and building risk modelling infrastructure 10. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and model developers confidence (akin to Price-Anderson Act for nuclear energy) 10. Clarification of legal liability to create clarity and account-ability (akin to Price-Anderson Act for nuclear energy) 11. Scale funding from $150m today to $500m+ annually for research into technical auditing of developers 12. Scale funding from $150m today to $500m+ annually for research into technical auditing of developers 13. Require adherence to a single industry standard

20. Industry leaders to provide new coverage with limits above $10bn for specific AI data center risks including weight exfiltration, and integrity against state-level threats 21. Government insurance mandates and backstops for catastrophic risk scenarios, to give insurers and data center developers confidence (akin to Price-Anderson Act for nuclear energy)

4. Auditors to build executive-level dashboards tracking real-time agent deployments and risks in the real world - similar to Basel III banking dashboards 5. Significant scale-up of technical auditing expertise and capacity from <1,00 deployments in 2024 to 1,000,000+ in 2027, powered by productionized testing and monitoring infrastructure building on examples like Inspect and Docent

6. Technical auditing expertise combining the rigor of PwC with the cutting edge AI capabilities of startups like Haize Labs to develop evaluations applicable to real-world harms, including multi-turn and multi-agent evaluations, and that robustly assess risk including using model internals and checking for sandbagging as needed. These tests are essential to ensure companies cannot cheat tests, like VW did with emissions

14. Auditors to build an ‘AI risk dashboard’ that can be shared with American intelligence as well as enterprises, tracking real world risks and evaluation techniques, e.g. like Iran’s use of OpenAI for an influence campaign

16. Auditing of organizational and system-level risks (as opposed to only model-level) to e.g. prevent state-affiliated misuse or information gathering. Build on best practices from defense, cyber, financial audits

22. Development of an auditable standard for Security-Level 4 and 5 building on existing frameworks and threat analyses

24. Audits that oversee the entire build process (e.g. tracking the chip supply chain to prevent espionage) alongside operations

25. Development of an auditable standard for Security-Level 4 and 5 building on existing frameworks and threat analyses

25. Develop- -ment of an auditable standard for Security-Level 4 and 5 building on existing frameworks and threat analyses

Building The Movement

This is merely a starting point. The fog around AI’s trajectory calls for a need to experiment with incentives quickly, fail, learn, and adapt. As evidenced by the rapid advances of AI research and application development, the stakes have never been higher, the timelines never more compressed. Now is the time to act.

Applying the incentive flywheel to underwrite secure AI progress needs the technologist’s ingenuity, the actuary’s carefulness, the business leader’s pragmatism, the economist’s incentive analysis, the legal scholar’s historical grounding and the researcher’s willingness to explore unusual futures.

The authors are building the incentive flywheel right now. If you are interested in contributing, reach out at rk@aiuc.com.

Footnotes

Table 2 below outlines examples across fire risk, car safety and AI.

Examples: Microsoft shut down their AI chatbot Tay in 2016 after it spewed racist and nazi ideology (link), Google’s Gemini outputted photos of people of color in nazi uniforms in 2024 (link), OpenAI rolled back a overly sycophantic version of ChatGPT in April 2025 (link), Google committed to publishing safety papers significant AI model releases, but shipped Gemini 2.5 Pro without the promised safety documentation (link)

High uncertainty over how much coverage and how it will be provided across industry, (re)insurers, and government. Nuclear energy sector in US has $16bn private coverage, airlines carry $2bn+ coverage per incident

Sandbagging refers to AI systems deliberately changing behaviour when they know they are being evaluated