Authors: Alexander Meulemans, Rajai Nasser, Maciej Wołczyk, Marissa A. Weis, Seijin Kobayashi, Blake Richards, Guillaume Lajoie, Angelika Steger, Marcus Hutter, James Manyika, Rif A. Saurous, João Sacramento, and Blaise Agüera y Arcas
Paper: https://arxiv.org/abs/2511.22226
WHAT was done? The authors introduce Embedded Universal Predictive Intelligence (MUPI), a comprehensive mathematical framework that redefines agents not as external observers acting on an environment, but as embedded entities within a joint universe. By constructing a Bayesian mixture over “universes” (programs defining joint agent-environment dynamics) rather than just environments, they solve the infinite recursion problem inherent in mutual prediction. They introduce the Reflective Universal Inductor (RUI)—a theoretical oracle ensuring the agent’s hypothesis class contains the agent itself—and define new game-theoretic solution concepts, such as the Subjective Embedded Equilibrium, which rationalize cooperation in scenarios like the Prisoner’s Dilemma where classical Nash equilibria fail.
WHY it matters? This work theoretically grounds the behavior of modern Foundation Models, which naturally predict sequences of interleaved actions and observations. It solves the long-standing “grain of truth” problem in multi-agent reinforcement learning (MARL), where an agent cannot perfectly model an environment that contains a copy of itself without infinite recursion. By proving that universal priors (Occam’s razor) naturally lead to structural similarity awareness, the paper provides a normative justification for Evidential Decision Theory in AI, suggesting that agents which model themselves as part of the world will naturally cooperate with structurally similar peers without needing explicit communication.
The foundational dogma of standard Reinforcement Learning (RL) is “decoupled agency.” In this dualistic view, the agent is a distinct entity separated from the environment by a hard interface of sensors and actuators (Figure 1.1).
The agent treats the environment dynamics as an external, often stationary, stochastic process ν(et∣x<t,at). While mathematically convenient for single-agent settings, this paradigm collapses in Multi-Agent RL (MARL). In MARL, “the environment” comprises other agents who are simultaneously learning and updating their policies. This induces non-stationarity, rendering retrospective learning (updating policies based on past data) fundamentally flawed.
To act optimally in such settings, an agent requires prospective learning: it must predict how other agents will update their behaviors. However, this leads to a “theory of mind” infinite regress. If Agent A models Agent B, it must account for the fact that Agent B is modeling Agent A modeling Agent B, and so on. Standard approaches to “co-player learning awareness” often cheat by assuming the opponent is a fixed learner, ignoring the recursion. This paper argues that the root cause of this failure is the decoupled perspective. To solve mutual prediction, an agent must model the universe—the joint distribution of its own actions and the environment’s responses—thereby treating its own policy as just another physical process within the world.
The core shift in MUPI is replacing the environmental conditional probability with a joint probability measure over the entire history, which the authors term a Universe. In standard Bayesian RL, an agent maintains a belief w(ν) over environments ν and predicts percepts e given actions a. In the embedded framework, the agent maintains beliefs w(λ) over universes λ. A universe λ:(A×E)∗→[0,1] is a probability measure over the sequence of both actions and percepts. Crucially, this means the agent predicts its own future actions. The predictive distribution ρ is defined as a Bayesian mixture:
This seemingly subtle shift has profound implications. In a decoupled setting, beliefs about the environment are updated only on percepts. In the embedded setting, beliefs are updated on actions as well. If an agent observes itself taking a specific action at, this eliminates all hypotheses λ in which the agent-part of the universe would not have taken at. This creates Coupled Beliefs: information about one’s own actions provides evidence about the nature of the external world (including other agents).
The central theoretical difficulty in embedded agency is the Grain of Truth (GoT) problem. For a Bayesian agent to converge to optimal behavior, the true universe μ must lie within its hypothesis class Muni (i.e., w(μ)>0). However, if the true universe contains the agent itself, and the agent is a complex Bayesian predictor using Muni, we enter a self-referential paradox. The agent’s complexity is necessarily greater than any single hypothesis in its class if that class is restricted to standard computable measures.
The authors solve this by introducing the Reflective Universal Inductor (RUI), building upon the foundational Reflective Oracles framework. They construct a class of universes described by Probabilistic Oracle Machines (POMs) that have access to a specific oracle τ. The oracle τ is defined “reflectively” to answer queries about the universal mixture ρτ constructed from those very machines. Specifically, a query takes the form ⟨b,p,h⟩, representing the question: “Is the probability ρτ(b∣h) greater than p?” The authors prove the existence of a fixed point (Theorem 5.16) where the oracle’s answers are consistent with the mixture distribution defined by machines having access to that oracle.
This effectively “closes the loop” as shown in Figure 5.1, allowing the definition of a hypothesis class MuniRUI that contains all computable universes and universes containing agents that reason using MuniRUI. This guarantees that an embedded agent using the RUI satisfies the Grain of Truth property: its predictions D∞(ρ,μ) converge to zero, ensuring consistent mutual prediction and infinite-order theory of mind.
The most behaviorally significant consequence of embedded agency is the formalization of Structural Similarity. Because beliefs are coupled, an agent’s policy π and the environment ν (containing other agents) are not independent variables; they share mutual information. This alters rational behavior in scenarios like the Twin Prisoner’s Dilemma, where an agent plays against an exact copy of its source code. In the classical decoupled/causal view (Nash Equilibrium), the agent treats its action as an independent intervention; regardless of the copy’s action, defecting yields a higher reward, so the agent defects. Conversely, under the embedded evidential view (MUPI), the agent recognizes that πyou=πopponent. Its decision to cooperate is strong evidence that the opponent (running the same code on the same input) will also cooperate. Therefore, the rational choice is to cooperate.
The authors formalize this via a new solution concept: the Subjective Embedded Equilibrium (SEE), which generalizes the Subjective Equilibrium of Kalai and Lehrer. An SEE describes a state where the agent acts optimally with respect to its subjective belief ρ, and this belief matches the ground truth μ on the play path. Because ρ allows for coupled beliefs—correlations between agent action and environment response not caused by direct physical causality—SEEs allow for cooperation in the Twin Prisoner’s Dilemma. The authors prove that embedded Bayesian agents satisfying the Grain of Truth property converge to playing an ϵ-SEE. They further define the Embedded Equilibrium (EE), the objective counterpart to the Nash Equilibrium. An EE explicitly incorporates a “dependency distribution” q that models the true structural constraints (e.g., aagent=aopponent). While Nash equilibria assume other agents’ policies are fixed regardless of your deviation, Embedded Equilibria account for the counterfactual: “If I were to deviate, the other agent (being similar to me) would likely deviate too.”
While the RUI is incomputable (serving as a theoretical gold standard), the authors describe practical approximations via k-step planner embedded agents. Similar to MuZero or Self-AIXI, these agents do not solve the infinite horizon planning problem perfectly. Instead, they perform a finite lookahead:
Here, 𝑄𝑘𝜉 represents the value estimated by unrolling the joint predictive model ρ for k steps. Crucially, the “rollout” uses ρ to predict both future percepts and the agent’s own future actions. This creates a form of self-learning awareness. As the agent learns and updates ρ, the Q-value estimates change. Because the agent uses ρ to predict its future self, it effectively anticipates its own future policy improvements. The authors prove (Theorem 4.31) that under certain conditions (”sensibly off-policy”), a group of k-step embedded planners converges to a Subjective Correlated Embedded Equilibrium.
Perhaps the most striking theoretical result is the justification for why agents should assume structural similarity. One might argue that assuming “other agents are like me” is an arbitrary bias. The authors refute this using Algorithmic Information Theory. They analyze the Solomonoff Prior, which assigns probability 2−K(λ) to a universe λ, where K is the Kolmogorov complexity (shortest program length). They prove (Theorem 5.36) that the Solomonoff prior is always unboundedly coupled.
Mathematically, if a policy π and environment ν share common subroutines (e.g., they both use a similar search algorithm or neural architecture), the joint complexity K(π,ν) is significantly lower than the sum of independent complexities K(π)+K(ν).
S(π,ν)≈K(π)+K(ν)−K(λ)≫0
Since the universal prior favors simpler explanations (Occam’s Razor), it assigns exponentially higher probability to universes where agents share structural similarities. Therefore, reasoning about structural similarities is not an ad-hoc assumption; it is a fundamental consequence of rational Bayesian inference in a computational universe.
Despite its theoretical elegance, the framework faces practical hurdles. First, the full Embedded AIXI agent and the RUI are not computable, serving primarily as idealizations to guide the design of practical approximations like neural sequence models. Any practical approximation using computational devices will likely violate the strict grain-of-truth property. Second, the convergence theorems rely on the prior assigning non-zero mass to the truth. In practice, agents might fall into “dogmatic traps”—believing a deviation from equilibrium leads to catastrophe (e.g., “if I stop defecting, the universe ends”)—and never exploring enough to disprove it. The paper acknowledges this requires active exploration strategies. Finally, the convergence of k-step planners relies on the “sensibly off-policy” condition, which assumes the value of the self-model’s policy doesn’t degrade catastrophically when taking off-policy actions. As noted in recent work on Joint AIXI (JAIXI), failures in embeddedness can lead to learning stagnation if this condition is unmet.
This paper fundamentally reframes the theory of multi-agent learning. By moving from Decoupled Agency (Agent vs. Environment) to Embedded Agency (Agent ∈ Universe), it resolves the infinite regress of mutual prediction and provides a rigorous formalism for “acausal” cooperation via structural similarity. The implications for modern AI are immediate. Large Language Models (LLMs) are trained on vast corpora of human interactions; they naturally model the joint distribution of dialogue (actions and percepts). This paper suggests that such “foundation model agents” are essentially approximations of Embedded Bayesian Agents. If these models are fine-tuned to maintain the full joint distribution rather than just a conditional policy, they may naturally exhibit the cooperative behaviors predicted by Evidential Decision Theory, leading to more robust coordination in mixed-motive settings than classical RL approaches could ever achieve.