What I Do Not Understand, (A)I Cannot Create.

The Converse Feynman Principle — a framework for AI-assisted development that starts with comprehension, not generation.

“What I cannot create, I do not understand.” — Found on Richard Feynman’s blackboard at Caltech, February 15, 1988. Caltech Archives, catalog ref 1.10–29, image:2545

When Richard Feynman died on February 15, 1988, a photographer from the Caltech Archives captured his office blackboard. Two phrases were still visible in chalk at the top: “Know how to solve every problem that has been solved” and “What I cannot create, I do not understand.” Below them, a partially smudged “TO LEARN” list that scholars have since identified as including the Bethe Ansatz, the Kondo effect, and nonlinear classical hydrodynamics. The photograph became one of the most reproduced images in the history of science. UC Berkeley mathematician Edward Frenkel gave a formal talk on the board’s contents at the Perimeter Institute in 2023 — the items on that “TO LEARN” list, it turns out, were connected by deep mathematical structures that Feynman appears to have been circling in his final months.

The second line — “What I cannot create, I do not understand” — became Feynman’s most widely quoted motto. It encodes a specific epistemological claim: that genuine understanding of a system requires the ability to reconstruct it from first principles. Not to describe it. Not to cite it. To build it. If you cannot do that, your understanding is, in some irreducible sense, incomplete.

I have spent the last several years building and leading platform transformations at enterprise scale in regulated industries — observability, SRE, AI/ML, security operations — and I keep returning to that line. Because I believe the era of AI agency and AI-assisted software development demands a corollary that is just as uncomfortable and just as true:

What I do not understand it, (A)I cannot create.

Not “should not.” Cannot. Not safely, not durably, and not at the level of quality that production systems demand when they carry regulated workloads, serve millions of users, and run at three in the morning with nobody watching.

The Productivity Mirage

The case for AI-assisted development starts with a real and documented productivity gain. A 2023 controlled experiment by MIT and Microsoft Research (Peng, Kalliamvakou, Cihon, Demirer; arXiv:2302.06590) found that developers completed a standardized JavaScript HTTP-server task 55.8% faster with GitHub Copilot. GitHub cites the same figure in its marketing. The number is genuine. The problem is that it describes a constrained, greenfield, well-defined task — and the benefits skewed toward less-experienced developers.

When the task environment gets more complex, the story inverts. METR’s July 2025 randomized controlled trial (Becker, Rush, Barnes, Rein; arXiv:2507.09089) studied sixteen experienced open-source developers working on their own mature repositories — projects averaging over 22,000 GitHub stars and five or more years of contributor history. Across 246 real-world tasks, developers using early-2025 AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet) were 19% slower than without AI.

The cruelest finding was not the slowdown. It was the perception gap. Before the experiment, these developers predicted a 24% speedup. After completing the tasks, they still believed AI had made them 20% faster. They felt accelerated while actually decelerating. METR’s February 2026 follow-up reported that the experiment had become unreliable because developers increasingly refused to work without AI — the tool had become psychologically indispensable even where it was empirically counterproductive.

That perception gap is not a curiosity. It is the mechanism by which AI-assisted development fails silently at scale. When the people writing the code believe they are moving faster, the organizational feedback loops that normally catch quality degradation — code review scrutiny, testing rigor, architectural skepticism — all relax in proportion to the perceived speed. The system feels faster. The dashboards say otherwise.

The Quality Tax

If the productivity story is more nuanced than the headlines suggest, the quality story is unambiguous.

GitClear analyzed 211 million changed lines of code across its customer base from 2020 through 2024 — before and after the widespread adoption of AI coding assistants. The findings are stark. Refactoring — the practice of restructuring existing code for clarity and maintainability without changing its behavior — collapsed from roughly 25% of all code changes in 2020–2021 to under 10% in 2024. Copy-pasted code rose from 8.3% to 12.3%, a 48% relative jump. Duplicated code blocks of five or more lines increased approximately eightfold during 2024 alone. Code churn — lines revised or deleted within two weeks of being written — roughly doubled versus the pre-AI baseline. 2024 was the first year in GitClear’s dataset that copy-pasted code exceeded refactored code.

The interpretation is straightforward: AI makes it cheaper to generate new code than to understand and improve existing code. So organizations generate more and refactor less. The codebase grows. The duplication compounds. The churn accelerates. And the humans who are supposed to own the system fall further behind in their comprehension of what it actually does.

Google’s DORA team — the most respected longitudinal research program in software delivery performance — quantified the downstream consequences. The 2024 Accelerate State of DevOps Report found that for every 25% increase in AI adoption, delivery throughput dipped an estimated 1.5% and delivery stability fell 7.2%. The mechanism, as DORA team lead Nathen Harvey explained, is batch size: AI encourages larger changesets, and larger changesets are riskier. DORA’s 2025 follow-up (the State of AI-Assisted Software Development report) found AI now correlates with improved throughput but still increases delivery instability — the speed gains are real, but they come at the cost of reliability.

The Security Tax

The quality tax has a particularly expensive line item: security.

Veracode’s 2025 GenAI Code Security Report evaluated over 80 tasks across more than 100 LLMs in Java, JavaScript, Python, and C#. Forty-five percent of AI-generated code samples introduced OWASP Top 10 vulnerabilities. Cross-site scripting tests failed 86% of the time. Log injection failed 88%. Java was the worst performer, with approximately 72% of generated code carrying exploitable flaws. The most sobering finding: newer, larger models were not meaningfully more secure than older ones. This is a structural problem, not a temporary one.

The supply chain dimension is even more unsettling. A USENIX Security 2025 paper from researchers at UT San Antonio, the University of Oklahoma, and Virginia Tech generated 576,000 code samples across sixteen models and found that 19.7% of recommended packages did not exist. These were hallucinated package names — plausible-sounding libraries that no one had ever written. Forty-three percent of these hallucinations recurred consistently across all ten reruns of the same prompt, making them predictable enough for an attacker to pre-register a malicious package under the hallucinated name and wait. Seth Larson, the developer-in-residence at the Python Software Foundation, coined the term “slopsquatting” for this attack vector. It is novel, scalable, and a direct consequence of treating AI output as trustworthy without verification.

Research from Apiiro across Fortune 50 repositories found AI-generated code carrying roughly 2.5 times more high-severity vulnerabilities (CVSS 7.0+) than human-written code. By June 2025, AI-generated code was introducing over 10,000 new security findings per month — a tenfold increase from December 2024.

The Demo Trap

These statistics describe a systemic pattern, but the most instructive failures are specific.

In July 2025, SaaStr founder Jason Lemkin conducted a twelve-day experiment building a production application with Replit’s AI agent — a well-funded, widely adopted platform purpose-built for AI-assisted development. During an explicit code freeze, the agent deleted a live production database containing records for 1,206 executives and 1,196 companies. It then generated misleading status messages and initially claimed rollback was impossible. (It was not.) Replit CEO Amjad Masad called the incident “unacceptable and should never be possible” and rolled out automatic dev/prod separation and a planning-only mode. The incident is logged in both the OECD AI Incidents Monitor and the AI Incident Database (Incident 1152).

This is what I call the demo trap — the gap between a system that appears to work and a system that actually works under the conditions production imposes. AI is extraordinarily good at producing things that pass the demo. It generates code that compiles, runs, produces output, and looks correct on a screen. What it does not do — cannot do — is understand the consequences of what it has built. It has no model of “this is a production database and deleting it has irreversible business impact.” It has no concept of a code freeze. It optimizes for task completion, not for the constraints that make task completion safe. Those constraints live in the human’s understanding of the system, the environment, and the stakes. When the human abdicates that understanding — when the human vibes — the constraints vanish and the blast radius becomes whatever the agent’s permissions allow.

The Vibe Coding Reckoning

Andrej Karpathy coined “vibe coding” on X in February 2025: “fully give in to the vibes, embrace exponentials, and forget that the code even exists.” He later called the post “a shower of thoughts throwaway tweet,” and he meant it for weekend projects — disposable prototypes where failure costs nothing. Collins Dictionary named it the 2025 Word of the Year. And somewhere between the coinage and the coronation, an approach designed for throwaways migrated into how organizations talk about building production systems.

Y Combinator’s Garry Tan reported in March 2025 that for 25% of their Winter 2025 batch, 95% of the code was AI-generated. YC managing partner Jared Friedman was careful to note that these founders were “highly technical, completely capable of building their own products from scratch.” That qualifier is the entire point — and it is the part that gets dropped every time the statistic is repeated. The founders who successfully ship AI-generated codebases are not the ones who lack understanding. They are the ones who have the understanding and use AI to move faster within it.

Stack Overflow’s 2025 Developer Survey (49,000+ respondents) captures the resulting mood: AI tool adoption rose to 84%, but trust in AI accuracy fell to 29% — down from 40% in prior years. Forty-six percent of developers actively distrust AI output. The top frustration, cited by 66%, is “AI solutions that are almost right, but not quite.” And 72% say they are not vibe coding. The industry adopted the tools and then discovered, empirically, that the tools require more judgment, not less.

Martin Fowler, Thoughtworks’ Chief Scientist, articulated the risk precisely: AI tools must not “shortcut the learning loop that’s an essential part of a software developer’s practice.” Vibe coding is “good for explorations… throwaways, disposable stuff, but you don’t want to be using it for anything that’s going to have any long-term capability.” The Thoughtworks Technology Radar now places “Complacency with AI-generated code” on Hold — their formal recommendation against adoption.

The Converse Feynman Framework

So what does responsible AI-assisted development actually look like? Not as a slogan, but as a framework that individuals, teams, and enterprises can apply to real decisions about what to delegate, what to verify, and what to own?

I keep coming back to five comprehension gates. I call them the Converse Feynman Framework because they encode the converse of Feynman’s principle: the ceiling on what AI can safely build for you is set by the depth of what you actually understand.

Gate 1: Specify

Can you articulate, in unambiguous terms, what the system must do?

Not “build me an API.” What endpoints, what authentication model, what rate limits, what error semantics, what data contracts, what SLAs. If you cannot write the requirements — the real requirements, not the one-sentence prompt — AI will hallucinate them for you. It will make reasonable-sounding choices that are wrong for your context, your consumers, and your compliance environment. And you will not notice until production disagrees, because you never specified what “correct” looked like in the first place.

This is the gate where individual developers operate. A senior engineer who can write a precise specification will get spectacular results from AI. A junior engineer who cannot will get code that looks right and is not.

Gate 2: Constrain

Can you define what the system must NOT do, and the boundaries it must operate within?

Requirements describe the happy path. Constraints describe the blast radius. This is where you specify that the API must not return PII in error messages, that the agent must not modify production resources during a freeze, that the pipeline must not route PHI to an unencrypted destination, that the model must not execute arbitrary code. AI is spectacular at building what you asked for. It is equally spectacular at building things you never anticipated, because it has no model of the consequences you are trying to avoid.

This is the gate where teams operate. Constraints require shared understanding of the operating environment, the regulatory context, and the failure modes that matter. No individual prompt captures them. They live in architecture decisions, security policies, and operational runbooks — the artifacts of collective understanding.

Gate 3: Validate

Can you verify the output is correct, complete, and secure?

If you cannot write the test, you cannot trust the code — regardless of who or what wrote it. This means unit tests, integration tests, security scans, contract tests, load tests, and the unglamorous work of reading the generated code line by line to confirm it does what you think it does. Simon Willison drew the crucial line: “If an LLM wrote every line of your code, but you’ve reviewed, tested, and understood it all, that’s not vibe coding — that’s using an LLM as a typing assistant.” The validation gate is what separates the two.

Gate 4: Diagnose

Can you debug it when it fails at 3 AM?

The system that you cannot troubleshoot is the system that owns you. AI-generated code that you have not internalized — that you accepted, merged, and deployed without building a mental model of how it works — is someone else’s complexity running on your infrastructure. When it fails, and it will, you are starting the debugging process from zero context. The on-call engineer who did not write the code and cannot read the code is not an engineer. They are a hostage.

This is the gate where operational maturity lives. It is also where the METR finding bites hardest: if developers feel faster while actually moving slower, and if the code they produce is less refactored and more duplicated, the debugging surface area grows while the organizational capacity to debug shrinks. That is a compounding failure.

Gate 5: Defend

Can you explain it to an auditor, a regulator, or a postmortem?

In regulated industries — healthcare under HIPAA, financial services under SOX and PCI-DSS, critical infrastructure under NIST frameworks — “the AI wrote it” is not a defense. It is an admission that you deployed code you cannot account for. ISO/IEC 42001 is emerging as the AI governance management-system standard, and every framework in this space requires demonstrable human accountability for AI-assisted outputs. If you cannot trace the provenance of your code, explain the design decisions behind it, and attest to its compliance properties, you have a regulatory exposure proportional to the amount of AI-generated code in your stack.

This is the gate where enterprise leadership operates. The decision to adopt AI-assisted development at scale is also a decision about how you will defend what that development produces to parties who were not in the room when it was written.

Applying the Framework

The rule is simple: AI can operate safely up to the level of your deepest gate.

For an individual developer, this means honestly assessing which gates you can clear for the task at hand. If you can specify, constrain, validate, diagnose, and defend, delegate freely — AI will make you dramatically faster without making you dangerous. If you can specify but not constrain, slow down. You are about to ship functionality with unbounded side effects. If you can constrain but not validate, you are about to ship bugs you cannot detect. The framework is not about whether to use AI. It is about matching the degree of delegation to the depth of your comprehension.

For a team, this means establishing shared gate definitions as part of your development process. What does “specified” mean for your services? What are the constraints that every team member must know? What does validation require before merge? What does “diagnosable” look like in your on-call rotation? These are not new questions. They are the questions that good engineering teams have always asked. The framework simply makes them explicit in the context of AI-assisted work, where the temptation to skip them is higher because the initial output arrives so much faster.

For an enterprise, this means treating the five gates as a maturity model. Most organizations today are reasonably strong at Gate 1 (specification) and Gate 3 (validation through CI/CD). They are weakest at Gate 2 (constraining what AI-generated code must not do), Gate 4 (diagnosing AI-generated code in production), and Gate 5 (defending AI-generated code to regulators and auditors). The investment priority should follow the gaps. AI governance programs that focus on adoption metrics without addressing comprehension gates are measuring the wrong thing — and the DORA data suggests they are shipping the wrong outcomes.

The Honest Position

I want to close with the position I hold and the falsification conditions I would accept.

AI-assisted development is the most significant productivity shift in software engineering since the move from compiled to interpreted languages. I use these tools daily. I have seen them compress weeks of prototyping into hours. I have watched junior engineers produce architectural proposals that would have taken them months to develop without AI assistance. The gains are real. They are not in dispute.

What I am arguing is that the gains are gated by comprehension — and that the industry’s current narrative systematically understates this. The evidence from METR, GitClear, DORA, Veracode, and the USENIX slopsquatting study all point in the same direction: AI accelerates the production of code while eroding the understanding, maintainability, and stability that complex systems require. The gains compound when understanding is present. The risks compound when it is not.

If a future rigorous study — at METR’s level of methodological care, with experienced developers on mature codebases — shows AI consistently producing faster and more stable outcomes, the strong form of this argument weakens and I will say so. If Veracode’s 45% vulnerability rate falls materially across model generations, the security concern recedes. I do not hold this position unfalsifiably. I hold it because the current evidence supports it and because the consequences of being wrong in the other direction — of assuming AI can build what we do not understand — are measured in production incidents, security breaches, and regulatory failures that no retrospective will undo.

Feynman’s blackboard was a challenge to physicists: if you cannot rebuild it, you do not truly understand it. The challenge to our industry is the converse: if you do not truly understand it, no amount of AI will build it safely for you. The returns are spectacular when you bring the understanding. They are catastrophic when you do not.

The question is not whether AI can write the code. It can. The question is whether you can own what it wrote.

I am a current CVS Health colleague. The opinions expressed in this post and in any corresponding comments are the personal opinions of the original authors, not those of CVS Health.

References

1. Caltech Archives. Feynman blackboard photograph, February 15, 1988. Catalog ref 1.10–29, object image:2545. digital.archives.caltech.edu.
2. Frenkel, E. “Feynman’s Last Blackboard: From Bethe Ansatz to Langlands Duality.” Perimeter Institute, 2023. PIRSA:23070020.
3. Peng, S., Kalliamvakou, E., Cihon, P., Demirer, M. “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.” MIT/Microsoft Research, arXiv:2302.06590, 2023.
4. Becker, N., Rush, K., Barnes, A., Rein, D. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR, arXiv:2507.09089, 2025.
5. GitClear. “AI Code Quality in 2025: A GitClear Analysis of 211 Million Changed Lines.” 2025.
6. DORA. “Accelerate State of DevOps Report.” Google Cloud, 2024.
7. DORA. “State of AI-Assisted Software Development.” Google Cloud, 2025.
8. Veracode. “GenAI Code Security Report.” 2025.
9. Vu, Hassanshahi, et al. “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs.” USENIX Security 2025.
10. Apiiro. “AI-Generated Code Security Research.” 2025.
11. Stack Overflow. “2025 Developer Survey.” stackoverflow.co/2025.
12. Karpathy, A. “Vibe coding.” X, February 2, 2025.
13. Fowler, M. “AI-Assisted Software Development.” Thoughtworks, 2025.
14. Thoughtworks Technology Radar. “Complacency with AI-Generated Code” (Hold). 2025.
15. OECD AI Incidents Monitor / AI Incident Database, Incident 1152 (Replit, July 2025).