Reinforcement Dreaming

Abstract

Most coding agents work in a straight line: plan, build, test, finish. If they make the wrong assumption early, they can waste hours going down the wrong path while still sounding confident about their progress.

We propose Reinforcement Dreaming (RD), a different way to run long-horizon agents. Instead of committing to one path, RD explores multiple executable futures of the same environment. We start with software repositories.

A Dream Run holds the goal, constraints, budget, and state of the task. From there, it can create separate candidate futures, or dreams, inside isolated sandboxes. Each dream can edit real code, run commands, test ideas, and record what actually happened.

The system decides where to spend more compute based on evidence, not on how confident the model sounds. It can keep pushing the main path, open a small side dream to answer a question, explore a completely different approach, combine useful findings, or return to an earlier checkpoint.

State is stored outside the model context, which lets a run continue for six to twelve hours even as the model's context is refreshed.

A candidate only becomes a Surviving Dream after a separate process verifies its claims, checks the important constraints, and records any remaining risks.

Within a single run, RD is an inference-time search and compute allocation system. It is not reinforcement learning. Over many runs, however, the evidence collected could be used to train better value models, improve how compute is allocated, and eventually fine-tune the system.

This paper describes the architecture, algorithms, and evaluation plan, along with the obvious ways it could fail: high cost, bad evaluators, differences between sandbox and production, or simply being beaten by a cheaper approach.

1. Introduction

Consider a concrete task: reduce the 95th-percentile latency of a service's request router by 30% without changing its public API.

A coding agent gets several hours to work on it. Early in the run, it decides serialization is the bottleneck and starts changing code around that theory. Three hours later, a benchmark points somewhere else: lock contention in the connection pool. By then, the context is full of serialization work, the summaries say progress is good, and the useful checkpoint from three hours ago is gone.

The problem is bigger than one bad guess. Long-running agents tend to commit to the first plausible path, mix their own narrative with what the environment actually proved, lose important state as contexts refresh, and keep working because a process is still running rather than because the work is still useful. Agent-computer interfaces made repositories practical environments for language-model agents [20], and execution-based benchmarks made their output easier to check [21], but most systems still push one main trajectory forward with retries.

Reinforcement Dreaming (RD) is an architecture for doing something different. It keeps several possible futures available and lets evidence decide which ones deserve more work.

Software repositories make this unusually practical. A worktree and its toolchain can be copied into an isolated sandbox where an agent edits real files, runs real commands, and sees real results without touching the original repository. Different theories about the task can therefore be tested in code instead of argued about in text.

A long run also needs an identity that survives the model working on it. RD stores the objective, constraints, budget, hypotheses, checkpoints, worktrees, open questions, and recorded outcomes outside any individual context. A model context becomes a temporary worker inside a larger Dream Run and can be refreshed without restarting the task.

Anything that touches the environment goes through an instrumented execution layer. Commands, results, commits, artifacts, benchmarks, and environment information are written to an append-only evidence ledger. Strategic decisions are then made from those records rather than from the model's description of its own progress.

Agent summaries are claims. Executed outcomes are evidence.

A Dream Run usually has one primary implementation. When the system reaches an important uncertainty, it can open a small shadow dream to answer one question, fork a larger alternative when the answer requires building both approaches, combine compatible findings, or return to a checkpoint. This creates diversity without paying the cost of running many fully independent agents for the entire task.

The final result also has to survive outside the context that produced it. Candidate work is re-executed in a clean environment, checked against the real objective, challenged by an independent reviewer, and passed through hard constraints. A candidate that clears those checks becomes a Surviving Dream, with its evidence and remaining risks attached. In the router example, that could be a branch that reproduces the requested 30% latency reduction. It could also be a useful negative result showing that serialization was tested and ruled out.

Within one Dream Run, RD is inference-time search with adaptive compute allocation. No model parameters change during the run. Across many runs, however, the ledgers produce structured experience that could later train value models, allocation policies, and eventually the underlying policy. Section 6 separates what exists today from that longer-term learning path.

This is an architecture paper and reports no benchmark results. The reference implementation already covers parts of the design, while Section 8 defines the experiment that would determine whether the extra machinery is actually worth using.

The main contributions are a concrete model for executable futures, persistent run state outside model context, evidence-guided branching and compute allocation, independent survival gates, and a path from long-running search toward learning across runs.

2. Definitions

We use dream as a technical term for a candidate future of the task. It makes no claim about cognition.

Isolated executable future

A forked repository worktree and toolchain running inside a sandbox. Commands and tests behave normally, but their effects stay inside the copy. The sandbox can reproduce much of the task environment, although it cannot guarantee the conditions of production traffic, hardware, external services, or the organization around the code.

Dream Run

The persistent process around the task, usually lasting six to twelve hours. It contains the objective, success criteria, constraints, deadline, compute budget, dreams, checkpoints, hypotheses, open questions, evidence, and surviving outputs. Many model contexts can work inside the same run.

Dream

A candidate trajectory attached to a hypothesis, checkpoint, worktree, history, and current estimate of usefulness. A dream can be active, shadow, merged, abandoned, or a candidate for survival.

Shadow dream

A small branch created to answer one question under a fixed budget. It might reproduce a failure, test a proposed explanation, benchmark a small alternative, review a design, or find the first bad commit. Its answer returns to the parent and the shadow ends.

Surviving Dream

A candidate that clears the pipeline in Section 5. Its important claims reproduce independently, hard constraints hold, and the result either improves the objective or proves a useful finding. Remaining risks travel with the result.

Evidence ledger

An append-only record of executed actions and outcomes, including commands, arguments, exit codes, commits, artifacts, parsed test or benchmark results, environment digests, timestamps, and the hypothesis changes supported by those records. Appendix B gives the minimum schema.

Policy

The mechanism that chooses what happens next from durable run state. In the current architecture, the policy combines a fixed language model with explicit orchestration rules and does not learn during a run.

Reward, value, survival, and the human objective serve different jobs. Local reward tells the controller that an action produced useful progress or information. Value estimates whether a dream deserves more compute. Survival determines whether a result cleared independent checks. The objective remains the request given by the human.

Collapsing these into one score creates an easy target for reward gaming [32]. An agent can improve a proxy by weakening a test or choosing a flattering benchmark, so the mechanism that explores the task should not also have final authority over what gets presented as correct.

3. Architecture and durable state

A long-running task has decisions happening at very different speeds. RD separates them into strategic, tactical, and execution control, borrowing the general shape of temporal abstraction [3] while using it as a systems boundary.

Strategic control operates over minutes to hours. It keeps track of the current theory of the problem, active dreams, major uncertainties, remaining budget, and evidence gathered so far. At review points it can continue the primary path, open a shadow, fork an alternative, combine findings, revert to a checkpoint, evaluate a candidate, or stop work on a dead path.

Tactical control works in minutes and owns the current subgoal inside one dream. It may reproduce a failure, establish a baseline, refactor behind existing tests, add a missing test, investigate a subsystem, or measure an implementation. It translates the strategy into a bounded piece of work and returns evidence or blockers.

Execution control works in seconds. It reads and edits files, runs commands, executes tests and benchmarks, and creates checkpoints. Actions that touch the environment are recorded through the instrumented execution layer.

This hierarchy also solves the context problem. The objective and constraints, worktrees, checkpoints, hypotheses, failed approaches, current strategy, evidence, open questions, and remaining budget live in durable run state. A model receives a rendered view of that state, does a piece of work, writes new events back, and can then be replaced. Algorithm 1 in Appendix A gives the loop that coordinates all of this.

A context refresh therefore does not depend on summarizing an entire conversation correctly. The new worker gets the current objective, constraints, subgoal, unresolved hypotheses, recent admissible evidence, relevant autopsies, and remaining budget. Old narration can disappear without erasing the run itself.

The evidence ledger is the factual history of that run. A useful record includes the exact command, exit code, commit, artifact hashes, parsed results where available, environment digest, and timestamp. A sentence saying that the tests passed may help another model understand what happened, but the executed test result is what downstream decisions can rely on.

This distinction matters more as runs get longer. Context compression can drop an old constraint or make a failed approach sound more promising than it was. An external record lets another model or a human trace a claim back to the environment that produced it and later provides cleaner data for cross-run learning than a collection of chat transcripts.

Time limits still exist, but they protect the system rather than decide whether an idea is good. Sandboxes have heartbeats and restart from checkpoints, workers refresh near their context limit, and the mechanisms in Table 1 bound the rest. Only stagnation evidence or a strategic review may end a dream.

A slow experiment can still be useful, while a busy agent can spend an hour generating no new information. Elapsed time settles neither question.

4. Allocation and progress

The central allocation problem is how to explore more than one possibility without turning every task into an expensive swarm.

Fully independent agents create diversity, but they also rediscover the same facts, build incompatible solutions, and multiply evaluator and coordination cost. A single continuous agent is cheaper and more coherent, but one early assumption can shape everything that follows. RD uses one primary implementation with temporary branches around it.

A shadow dream exists to buy information. Every shadow must name the question it is trying to answer, the decision that answer could change, its budget, and the artifact it must return. A full alternative is justified only when the uncertainty cannot be settled cheaply and both strategies need to be built far enough to compare.

Dreams share what they learn. If one experiment rules out a hypothesis, that evidence can affect every active path. Compatible findings can merge into the primary, while abandoned paths leave short autopsies so the same mistake does not immediately reappear.

At each strategic review, the controller decides where the next unit of compute is most useful. The reference policy uses the heuristic

score(a) = (ΔÔ(a) + λ Î(a)) / ĉ(a) · ρ(a)(1)

where ΔÔ estimates verified progress toward the objective, Î estimates information that could change a pending decision, ĉ estimates cost, and ρ discounts risky or hard-to-reverse actions. The coefficient λ sets the price the run is willing to pay for information. A small diversity bonus can break close decisions in favor of an approach meaningfully different from what has already been tried [16].

Equation 1 is a heuristic, not an optimal policy. Early estimates come from the model and are likely to be badly calibrated. When the available choices have little evidence separating them, simpler racing methods drawn from best-arm identification, such as successive halving or Hyperband, remain reasonable fallbacks [11] [12] [13] [14].

Progress is also broader than immediately improving the final metric. On a long software task, a benchmark may stay flat until near the end, while several earlier discoveries still make the solution much more likely. Establishing a reliable baseline, making a flaky failure reproducible, ruling out a plausible diagnosis, adding a test that changes a decision, finding the first bad commit, isolating a race, or proving an approach unworkable can all count as useful progress.

The common requirement is that the event is verified and changes a live decision. Surprise alone does not earn more compute, which is the software form of the noisy-TV problem in curiosity-driven exploration [18]. A flaky test can produce endless novelty, a dependency tree can always be explored one layer deeper, and a refactor can always create more work. Shadows remain bounded because the system values uncertainty reduction only when it affects something the run is currently deciding.

When a dream stops producing useful evidence, the controller intervenes. It can challenge the current assumptions, ask an independent critic to inspect the evidence, generate rival hypotheses, refresh the worker, return to a checkpoint, or open a shadow against the blocker. Repeated failure after those interventions can end the path and produce an autopsy. Appendix A defines the specific stagnation signals and intervention logic.

5. Survival

The process that builds a candidate does not control whether that candidate survives. Evaluation happens outside the dream and starts by recreating the work in a clean environment.

First, the evaluator clones the candidate commit into a sandbox the acting dream never touched. Claimed builds, tests, and benchmarks are replayed from their recorded evidence, and comparative measurements are run against the relevant base commit. When a measurement is noisy, repeated runs estimate its variance and that uncertainty stays attached to the result.

The candidate is then reviewed against the actual objective. For the router task, improving some convenient microbenchmark does not satisfy a request to reduce p95 latency on the specified request path. A model or human reviewer can help identify that mismatch, but generative review is itself noisy and gameable [33]. Its score may annotate or order candidates; it cannot act as a gate on its own.

A separate adversarial pass tries to break the result. It can add edge-case tests, look for weakened assertions, compare the suite with the base commit, inspect coverage, and check whether benchmark selection made the change look better than it is.

Finally, hard constraints are applied. The project must build, the base test suite must not regress, protected paths cannot be changed, security and license checks must pass, and user-declared invariants remain in force. A strong performance result does not compensate for breaking one of those constraints.

A candidate that clears the pipeline leaves with the change itself, evidence supporting its claims, measured deltas and uncertainty, and a disclosure of environmental assumptions, untested paths, and remaining work. A human still decides whether to adopt it.

The result also does not have to be a patch. A run that proves a major hypothesis wrong can return that finding with the experiment that ruled it out. In the router example, knowing that serialization is not responsible for the latency may be worth more than another speculative implementation.

6. From search to reinforcement

Within a single Dream Run, the system adapts by changing its state and reallocating compute as evidence arrives. The model itself does not learn. The longer-term idea begins when many completed runs create a dataset of repository states, actions, costs, verified outcomes, survival decisions, and human adoption.

We describe that path in levels so the claims remain clear.

L0, no memory. Every run starts cold. This is the baseline that later forms of memory have to beat.

L1, retrieval memory. Completed ledgers produce short autopsies, useful maneuvers, and repository facts that later runs can retrieve as untrusted context. Parameters stay fixed; behavior changes because the context improves. This follows prior work on memory and skill accumulation in agents [22] [23] [27] [28] [29]. L1 is partly implemented in mouse.dev.

L2, statistical priors. Simple models fit accumulated run data to estimate things such as action cost, which uncertainties tend to justify branches, or which tests tend to be informative. These parameters describe the environment and help the controller make better estimates. This level is specified but not implemented.

L3, learned decision models. A model trained to predict survival from dream state becomes a value function. An allocation policy trained from states, decisions, verified outcomes, and costs begins to learn persistent decision parameters across runs. At this point, describing part of the system as reinforcement learning becomes technically reasonable. This level is specified but has not been built.

L4, policy fine-tuning. The underlying model itself trains on trajectories with verified outcomes, extending ideas from reinforcement learning with verifiable rewards [1]. Human adoption could add preference information [34]. This remains speculative, and no such training has been run.

The evidence format is designed so these datasets can emerge from normal operation rather than requiring a separate logging project later. Whether learning from them is worthwhile remains an empirical question.

Cross-run learning also creates new failure modes. A lucky strategy can become an unjustified prior, one repository can be overrepresented, and a learned proxy can drift away from what the human actually asked for [30] [31]. Retrieved experience therefore still has to survive current evidence, while evaluators and hard gates remain outside learned components.

A further extension would let RD work on parts of its own controller or evaluator. Systems such as the Darwin Gödel Machine and self-improving coding agents explore related territory [24] [25]. We treat this as a separate, higher-risk direction rather than part of the core architecture.

7. Implementation and control

The reference implementation has four main pieces: the mouse.dev client, the run controller, isolated execution sandboxes, and an evaluator outside the acting dreams.

The client is the human surface. A person starts a Dream Run with an objective, constraints, and budget, then receives surviving outputs with evidence and disclosed risks. The run controller owns durable state, manages sandbox lifecycles, moves work between control levels, and performs the strategic loop described in Algorithm 1.

Dreams run in isolated microVMs containing forked worktrees and toolchain snapshots. Sandboxes do not contain credentials for the canonical repository or production systems, network access follows an allowlist, and compute and spending have hard ceilings. Tool permissions and protected paths are enforced outside the acting model, while checkpoints keep interventions reversible.

Long autonomous runs need controls that keep working even when the agent behaves badly. Commands pass through the instrumented execution layer, destructive patterns can be blocked, actions remain auditable, and a run can be terminated while preserving its state. No candidate merges into the canonical repository during dreaming, and deployment remains a human decision.

At the time of writing, sandboxed dream execution, the evidence ledger, checkpointing, survival gates, and L1 retrieval are implemented. Hierarchical allocation and shadow dreams are partial. L2 through L4 remain specifications.

8. Experimental design

No empirical results are reported in this paper. The point of this protocol is to test whether RD improves long-horizon agent work enough to justify its added cost and complexity.

Tasks and baselines

The evaluation uses two groups of tasks. The first draws repository issues from execution-checked benchmarks in the SWE-bench family [21], restricted to problems that take a strong linear agent more than thirty minutes so the long-horizon machinery has time to matter. The second group is built specifically around longer objectives: measurable performance improvements, dependency migrations with compatibility constraints, and diagnosis problems with hidden ground truth such as an injected first bad commit or a seeded race.

All systems remain blind to hidden acceptance tests and receive matched compute budgets. The baselines are:

1. A single long-running linear agent with retries.
2. Best-of-N independent agents with execution filtering [10].
3. A fixed planner, coder, and reviewer pipeline with no adaptive allocation.
4. A simple retry loop.
5. RD with uniform allocation replacing the adaptive policy.
6. A human-directed coding agent as an upper-cost reference.

Metrics

• Task success on hidden tests.
• Reproducibility of claimed results under independent re-execution.
• Regression rate, wall-clock time, and dollar cost.
• Decision-relevant findings per run.
• Calibration of stated confidence against outcomes.
• Human review burden, measured as time to adopt or reject.
• Robustness to a deliberately misleading first hypothesis.
• Coherence across forced context refreshes.
• Marginal value of shadow dreams, measured by ablation.

Ablations

Each ablation removes one component and replaces it with a simpler mechanism.

• Evidence ledger, replaced by model summaries.
• Independent evaluator, replaced by self-grading.
• Branching, replaced by a primary-only run.
• Shadows, replaced by full branches.
• Context refresh, replaced by one context until exhaustion.
• Survival gates, replaced by score-based selection.
• Cross-run memory, replaced by no memory.
• Adaptive allocation, replaced by a fixed schedule.
• Hierarchical control, replaced by one flat stream.

What would count against RD

The architecture should lose the argument if the complexity does not buy anything. In matched-compute experiments, evidence against RD would include best-of-N matching or beating its task success, ablated versions performing as well as the full system, evaluator decisions failing to predict human adoption, supposedly different dreams repeatedly exploring the same hypothesis, external state failing to improve coherence across context refreshes, or cross-run memory helping only on the repository where it was collected.

Those are failures of the central claim, not implementation details to explain away. The results should be reported either way.

10. Limitations and failure modes

The largest limitation is simple: there is no evidence yet that RD works better than cheaper approaches. Until the protocol in Section 8 runs, the architecture's central performance claim remains a hypothesis.

The sandbox is another hard boundary. A repository copy and toolchain cannot recreate production traffic, hardware differences, external services, hidden security context, team constraints, or years of future maintenance. Independent reproduction can show that a result survives the test environment; it cannot prove that deployment is safe.

Compute may also make the entire approach unattractive. Maintaining alternative futures costs time and money, and the orchestration itself adds overhead. If a linear agent, best-of-N search, or a simpler engineering process reaches the same outcomes under the same budget, the simpler system wins.

Diversity is imperfect because dreams can share the same model and therefore the same blind spots. Several branches may independently arrive at the same wrong idea. Independent critics and different contexts can help, but they do not turn correlated models into independent sources of truth.

The evaluator is also fallible. Tests rarely specify everything that matters, benchmarks can reward the wrong behavior, and generative reviewers remain noisy [33]. An agent repeatedly optimizing against the same gates may eventually find weaknesses in those gates [32], even when adversarial testing and suite checks make that harder.

Branching can become its own failure mode. New futures created faster than useful evidence arrives multiply cost without adding information, while independently developed implementations may be too different to merge cleanly. In some cases, rebuilding one solution from the useful findings will be cheaper than combining branches.

Durable state preserves mistakes as well as useful information. A subtly wrong interpretation of the objective can survive across every context refresh and guide a ten-hour run in the wrong direction. External memory solves forgetting; it does not solve misunderstanding.

Finally, infrastructure consumes part of the same budget. Sandboxes fail, dependencies disappear, registries go down, benchmarks become flaky, and toolchains break for reasons unrelated to the agent. The evidence ledger can make those failures easier to diagnose, but it cannot recover the time they consumed.

These are reasons to test the architecture aggressively rather than reasons to expect every problem to be solved by adding more agents or more compute.

11. Conclusion

Long-running coding agents have a basic problem: one early theory can shape hours of work, while the model's story about its progress slowly becomes harder to separate from what the repository actually proved.

Reinforcement Dreaming keeps more than one future available. It stores the run outside any single model context, records executed evidence, spends additional compute when that evidence justifies it, and independently checks a candidate before presenting the result to a human.

Inside one run, this is search. Across many runs, the resulting evidence could eventually support learned value models, allocation policies, and model training.

Whether any of that is worth the cost is still unanswered. The architecture matters only if the experiments show that it solves difficult long-horizon tasks more reliably than simpler ways of spending the same compute.

References

1. K. P. Murphy. "Reinforcement learning: An overview." arXiv preprint arXiv:2412.05265, 2024.
2. R. S. Sutton. "Dyna, an integrated architecture for learning, planning, and reacting." ACM SIGART Bulletin, 2(4):160-163, 1991.
3. R. S. Sutton, D. Precup, and S. Singh. "Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning." Artificial Intelligence, 112(1-2):181-211, 1999.
4. D. Ha and J. Schmidhuber. "World models." arXiv preprint arXiv:1803.10122, 2018.
5. D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. "Dream to control: Learning behaviors by latent imagination." ICLR, 2020.
6. J. Schrittwieser, I. Antonoglou, T. Hubert, et al. "Mastering Atari, Go, chess and shogi by planning with a learned model." Nature, 588:604-609, 2020.
7. C. Browne, E. Powley, D. Whitehouse, et al. "A survey of Monte Carlo tree search methods." IEEE Transactions on Computational Intelligence and AI in Games, 4(1):1-43, 2012.
8. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. "Tree of Thoughts: Deliberate problem solving with large language models." NeurIPS, 2023.
9. C. Snell, J. Lee, K. Xu, and A. Kumar. "Scaling LLM test-time compute optimally can be more effective than scaling model parameters." arXiv preprint arXiv:2408.03314, 2024.
10. Y. Li, D. Choi, J. Chung, et al. "Competition-level code generation with AlphaCode." Science, 378(6624):1092-1097, 2022.
11. J.-Y. Audibert, S. Bubeck, and R. Munos. "Best arm identification in multi-armed bandits." COLT, 2010.
12. Z. Karnin, T. Koren, and O. Somekh. "Almost optimal exploration in multi-armed bandits." ICML, 2013.
13. K. Jamieson and A. Talwalkar. "Non-stochastic best arm identification and hyperparameter optimization." AISTATS, 2016.
14. L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. "Hyperband: A novel bandit-based approach to hyperparameter optimization." JMLR, 18(185):1-52, 2018.
15. M. Jaderberg, V. Dalibard, S. Osindero, et al. "Population based training of neural networks." arXiv preprint arXiv:1711.09846, 2017.
16. J.-B. Mouret and J. Clune. "Illuminating search spaces by mapping elites." arXiv preprint arXiv:1504.04909, 2015.
17. A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. "First return, then explore." Nature, 590:580-586, 2021.
18. Y. Burda, H. Edwards, A. Storkey, and O. Klimov. "Exploration by random network distillation." ICLR, 2019.
19. C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. "GenProg: A generic method for automatic software repair." IEEE Transactions on Software Engineering, 38(1):54-72, 2012.
20. J. Yang, C. E. Jimenez, A. Wettig, et al. "SWE-agent: Agent-computer interfaces enable automated software engineering." NeurIPS, 2024.
21. C. E. Jimenez, J. Yang, A. Wettig, et al. "SWE-bench: Can language models resolve real-world GitHub issues?" ICLR, 2024.
22. N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. "Reflexion: Language agents with verbal reinforcement learning." NeurIPS, 2023.
23. G. Wang, Y. Xie, J. Jiang, et al. "Voyager: An open-ended embodied agent with large language models." arXiv preprint arXiv:2305.16291, 2023.
24. J. Zhang, S. Hu, C. Lu, R. Lange, and J. Clune. "Darwin Gödel Machine: Open-ended evolution of self-improving agents." arXiv preprint arXiv:2505.22954; ICLR 2026.
25. M. Robeyns, M. Szummer, and L. Aitchison. "A self-improving coding agent." arXiv preprint arXiv:2504.15228, 2025.
26. A. Novikov, N. Vũ, M. Eisenberger, et al. "AlphaEvolve: A coding agent for scientific and algorithmic discovery." arXiv preprint arXiv:2506.13131, 2025.
27. S. Zhang, J. Wang, R. Zhou, et al. "MemRL: Self-evolving agents via runtime reinforcement learning on episodic memory." arXiv preprint arXiv:2601.03192, 2026.
28. P. Xia, J. Chen, H. Wang, et al. "SkillRL: Evolving agents via recursive skill-augmented reinforcement learning." arXiv preprint arXiv:2602.08234, 2026.
29. Z. Cao, J. Deng, L. Yu, et al. "Remember me, refine me: A dynamic procedural memory framework for experience-driven agent evolution." arXiv preprint arXiv:2512.10696, 2026.
30. J. Fang et al. "A survey of self-evolving agents: What, when, how, and where to evolve." arXiv preprint arXiv:2507.21046, 2025.
31. S. Shao, Q. Ren, C. Qian, et al. "Your agent may misevolve: Emergent risks in self-evolving LLM agents." ICLR, 2026.
32. J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger. "Defining and characterizing reward gaming." NeurIPS, 2022.
33. L. Zheng, W.-L. Chiang, Y. Sheng, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." NeurIPS, 2023.
34. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. "Deep reinforcement learning from human preferences." NeurIPS, 2017.

Appendix A. Component algorithms

Appendix B. Evidence record schema

This is the minimum admissible record. An asterisk marks a required field.

{
  "id"*:          "evt_01J...",
  "run_id"*:      "run_...",
  "dream_id"*:    "dream_...",
  "t"*:           "2026-07-21T03:14:07Z",
  "actor"*:       "execution|tactical|eval",
  "cmd"*:         "pytest -q tests/router",
  "args":         ["--maxfail=1"],
  "exit"*:        0,
  "commit"*:      "9f3c1e...",
  "artifacts": [
    {
      "path": "...",
      "sha256": "..."
    }
  ],
  "parsed": {
    "tests": {
      "passed": 214,
      "failed": 0
    }
  },
  "env_digest"*:  "sha256:...",
  "duration_ms"*: 41830,
  "admissible"*:  true,
  "belief_updates": [
    {
      "hypothesis": "H2-serialization",
      "status": "eliminated",
      "basis": "evt_01J..."
    }
  ]
}

Free-text narration and model self-assessments use companion records with admissible=false and no parsed field. Strategic decisions, value estimates, and survival evaluation may cite only admissible records.