In RSI, The Map is not the Territory

Welcome to Brain-Inspired AI!

I’m back! I am Bill Lambos, the same human being who brought you the Lamb_OS Substack. My new book, Brain-Inspired Artificial Intelligence (or “BIAI”), is an anchor for this Substack, and I’m humbled that it’s getting such excellent reviews. BIAI tells the story of why transformer models, or GPTs - no matter how exceptional - are fundamentally misaligned with biological intelligence. They offer no path to AGI, and as such, they are now more a part of AI’s past than of its future. Check out the free landing page for the book: See for yourself at BIAI.

This post addresses a fundamental fact about transformer models: they have what therapists like to call “issues with reality.” In short, AI is unreliable. The causes are multiple - non-representative, stale or insufficient training data, errors in reasoning, hallucinations, failures to maintain context, and manipulative prompting. These are serious issues that continue to render AI output as suspect. To remediate this, some labs and companies have suggested they can get around AI’s problems with reality testing by having AI train itself, a technique called Recursive Self-Improvement (or RSI). I’m here to advise you that this is not going to work.

The concept is attractive both for its simplicity and for the problems it could solve: build a model capable of improving its own performance, based on its own track record. Then retain the benefits by having the model apply the improvements to itself. Each “generation of enhancement” is expected to yield not only a (marginally) more capable system, but a lasting improvement to the model itself. There has been a fair amount of chatter in the last year regarding recursive self-improvement (RSI), with some companies claiming this strategy may produce intelligence that exceeds human capabilities. Thus, RSI has emerged as a prominent candidate for post-transformer development.

However, those of us who study biological intelligence ask you consider a rather different opinion. Animal brains boast of being the only known system that achieves genuine self-improvement over time. They do so by operating under a constraint typically omitted in RSI: continuous validation against the real world. This distinction is not a minor detail of implementation; it is central to the possible efficacy of RSI..

Two Forms of Self-Improvement

The difference in opinions requires distinguishing these two approaches. In the transformer paradigm, RSI typically involves one of three mechanisms:

1. A model generates synthetic outputs intended to serve as training data for subsequent learning iterations;

2. A model evaluates and revises its own results;

3. More ambitiously, an agent modifies its own code and architecture.

In each scenario, the primary source of new information is internal to the system.

Biological systems learn continuously, following a different principle. Cortical circuits continuously predict incoming stimuli; the discrepancy between prediction and observation—the prediction error—drives synaptic adaptation. Although the brain also processes internal activity, particularly during sleep, this activity is anchored in waking experience and is continually reconciled with new sensory input. The biological loop traverses the external world, whereas the RSI loop operates within a closed system.

Internal Dynamics of the Closed Loop

The most definitive evidence regarding the consequences of recursive training in learning systems is offered by Shumailov et al. in Nature (vol. 631, July 2024). By training successive generations of language models exclusively on the outputs of their predecessors, the researchers observed that “indiscriminate learning from model-generated data induces ‘model collapse’—a destructive process in which models progressively lose fidelity to the true underlying data distribution.” (Emphasis mine). This effect was replicated in several different Gaussian mixture models and variational auto-encoders, prompting the authors to characterize it as inherent to generative models trained recursively on prior outputs (Nature).

It is the inherent functionality of deep learning networks that makes them inaccessible to any simple fix. Collapse originates in the tails of the data distribution: outliers and rare events, atypical sequences of tokens, and minority cases will become sampled less frequently. This in turn leads each generation to underrepresent them, until they disappear. Eventually, models converge on a narrow distribution “with substantially reduced variance” that diverges markedly from the original (Nature). The authors attribute this to three compounding factors:

1. Statistical approximation error arising from finite sampling,

2. Functional expressivity error due to model limitations, and

3. Functional approximation error stemming from imperfect learning procedures.

The authors’ supplementary analysis demonstrates that even when every generation is free from expectation bias, the divergence from the original distribution may grow exponentially, because “errors accumulate over time” (University of Oxford repository).

This represents error propagation in its most basic presentation. No individual iteration is catastrophic; in fact, each iteration is nearly faithful. However, cumulative “near fidelity” results in a stochastic drift away from reality, favoring homogeneity and typicality.

The Rebuttal, its Concessions, and the Exception that Proves the Rule

It may not surprise you to hear that disagreement exists with the above view. Gerstgrasser et al. (2024) demonstrated that model collapse is contingent on a specific procedure: substituting synthetic data for real data in each generation. When synthetic data accumulates alongside original human data, test error remains bounded across iterations. A subsequent study, “Collapse or Thrive?”, corroborated this pattern across multiple contexts and indicated that synthetic data’s utility depends on the volume of underlying real data.

While this is good to know, it acknowledges a critical limitation of RSI which is inherent to ANNs: collapse is averted only because the original, externally grounded data is retained and continues to anchor each generation. The stabilization arises from the access to the real world rather than from recursion per se.. The accumulation of fresh real world exemplars limits degradation, but it cannot transform the loop into a mechanism for generating new knowledge about reality. A system increasingly dependent on a fixed reservoir of human data is not achieving recursive improvement; rather, it is reinterpreting a static corpus with expanding marginal annotations.

Self-Correction Without External Feedback

An analogous result is seen when models attempt to enhance reasoning rather than training data. Google DeepMind team investigated intrinsic self-correction (where a model revises its output using only internal judgment) and reported that “LLMs struggle to self-correct their responses without external feedback, and at times, their performance might even degrade post self-correction” (Huang et al.). On the GSM8K mathematics benchmark, when models altered their responses, they were more likely to convert correct answers into incorrect ones than to rectify errors (Huang et al.). Related research identified error detection as the primary bottleneck: models have difficulty identifying their own logical flaws, although they can correct them once the error is specified (Tyen et al., ACL Findings 2024). A system incapable of reliably detecting its own errors cannot reliably improve itself. Each iteration is as likely to entrench an error as to eliminate one.

The ability of animal brain to continuously learn without either knowledge collapse or or catastrophic forgetting presents a salient and contrasting challenge to the RSI-based future proposed by frontier labs. As discussed in BIAI, the brain is not merely a larger transformer; it is a distinct learning mechanism operating at approximately 20 watts, without pauses for retraining or access to curated datasets. It addresses at least three issues encountered in RSI:

1. Grounding. Every waking moment generates new prediction errors. A mistaken assumption about whether a step will support one’s weight is corrected by the step itself. Biological learning constitutes self-improvement in that the organism directs attention, action, and exploration, but the evaluation of each update is provided by reality rather than the learner.

2. Consolidation without “erosion”. The hippocampus rapidly encodes episodic information, and during sleep, these traces are replayed and progressively integrated into neocortical knowledge. This process represents a biological analog of learning from internally generated data that avoids collapse. Computational modeling provides insight: in a thalamocortical framework, sleep-related replay of both older and newer memory traces reversed the interference caused by new learning by reorganizing synapses to permit coexistence of competing memories (González et al.). When this principle was applied to artificial networks via a sleep-like phase employing local Hebbian plasticity, it restored performance on tasks previously subject to catastrophic forgetting (Tadros et al., Nature Communications 2022). Internal replay remains stable in the brain because it is interleaved with and constrained by waking experience.

3. Preserved plasticity. Conventional deep learning exhibits a subtle failure mode beyond forgetting. Dohare, Sutton, and colleagues (Nature, 2024) demonstrated that continuously trained networks progressively lose learning capacity until performance matches that of shallow architectures, as units become “dormant, overcommitted, and homogeneous.” Their proposed solution, continual backpropagation, reinitializes a subset of underutilized units; they conclude that gradient descent alone is insufficient: sustained learning “requires a random, non-gradient component to maintain variability and plasticity” (PMC). The brain incorporates such mechanisms intrinsically: synaptic turnover, neuromodulatory regulation via dopamine and acetylcholine, homeostatic scaling, and, in certain regions, neurogenesis.

Viewed collectively, model collapse and loss of plasticity represent the same pathology, but seen standing on distinct perches. Collapse reflects diminished diversity in model output; loss of plasticity reflects diminished diversity in the learning apparatus. Recurrent self-improvement (RSI), by feeding a system its own output with a fixed optimizer, tends to exacerbate both conditions.

Implications for the post-transformer era

The utility of synthetic data, self-critique, and self-modifying agents is not in question; they have been shown to be valuable instruments associated with measurable progress in verifiable domains such as programming and mathematics.

However, and unfortunately, the overarching and enduring assertion is that recursion does not generate new information. A closed loop may refine, compress, or reorganize existing knowledge, but it cannot reliably identify unknown information; over successive iterations, minor errors tend to accumulate more rapidly than corrections. The propagation of error is, by definition, an exponential function. Model collapse is therefore also exponential; sooner than later, the original training data have been corrupted, and the model generates nothing other than nonsense.

Consequently, recursive self-improvement (RSI) is an unlikely successor to the transformer paradigm. It inherits the transformer’s fundamental limitation—a static relationship with the external world mediated by a fixed corpus of training data—and exacerbates it. Biological intelligence suggests an alternative direction: systems that learn continuously from prediction error, act to acquire evidence, consolidate knowledge offline without discarding rare experiences, and maintain flexible learning mechanisms.

The future does not lie in models that intensify self-reflection; it lies in models that remain oriented toward external observation and the detection of error.

References

1. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). AI models collapse when trained on recursively generated data. Nature, 631, 755–759. https://www.nature.com/articles/s41586-024-07566-y

2. Gerstgrasser, M., Schaeffer, R., et al. (2024). Is model collapse inevitable? Breaking the curse of recursion by accumulating real and synthetic data. arXiv:2404.01413. https://arxiv.org/abs/2404.01413

3. Kazdan, J., Schaeffer, R., et al. (2024). Collapse or thrive? Perils and promises of synthetic data in a self-generating world. arXiv:2410.16713. https://arxiv.org/html/2410.16713v1

4. Huang, J., Chen, X., Mishra, S., et al. (2024). Large language models cannot self-correct reasoning yet. ICLR 2024. https://arxiv.org/html/2310.01798v1

5. Tyen, G., Mansoor, H., Cărbune, V., Chen, P., & Mak, T. (2024). LLMs cannot find reasoning errors, but can correct them given the error location. Findings of ACL 2024. https://aclanthology.org/2024.findings-acl.826.pdf

6. Zhang, J., Hu, S., Lu, C., Lange, R., & Clune, J. (2025). Darwin Gödel Machine: Open-ended evolution of self-improving agents. arXiv:2505.22954. https://arxiv.org/abs/2505.22954

7. Wang, W., Piękos, P., et al. (2025). Huxley-Gödel Machine. arXiv:2510.21614. https://arxiv.org/html/2510.21614v2

8. Dohare, S., Hernandez-Garcia, J. F., Lan, Q., Rahman, P., Mahmood, A. R., & Sutton, R. S. (2024). Loss of plasticity in deep continual learning. Nature, 632, 768–774. https://pmc.ncbi.nlm.nih.gov/articles/PMC11338828/

9. Tadros, T., Krishnan, G. P., Ramyaa, R., & Bazhenov, M. (2022). Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks. Nature Communications, 13, 7742. https://www.nature.com/articles/s41467-022-34938-7

10. González, O. C., Sokolov, Y., Krishnan, G. P., Delanois, J. E., & Bazhenov, M. (2020). Can sleep protect memories from catastrophic forgetting? eLife, 9, e51005. https://pmc.ncbi.nlm.nih.gov/articles/PMC7440920/