· Jasper Gilley
Reward optimization chooses among the answers pretraining already taught, and the more ways there are to be right, the more that choosing costs.
RL from a blank slate conquers games. On language we pretrain first and use reward only to fine-tune at the end. Here is what reward optimization actually does to a model whose generating process we know exactly — three mechanisms, five levels of synonymy, two verifiers, and a boundary that never moves. Scroll
RL from a blank slate conquers games. On language we pretrain first and use reward only to fine-tune at the end. One major difference between the two: how many right answers each state admits.
The sharper question
Call the set of rules pretraining actually taught the model its basis: everything it can produce at all. Does reward optimization add to the basis, or only rearrange and degrade what is already there?
The Random Hierarchy Model: a grammar over 8 symbols. Each symbol rewrites into 2 children by one of \(m\) rules, and every rule is a different way of saying the same thing. Six levels down, a sequence of 64 tokens.
\(m{=}1\): one way to say everything, so the whole language is 8 sequences. A game. \(m{=}8\): every pairing of children is some rule, so every sequence is grammatical. Noise.
Sweep \(m\in\{1,2,3,4,6\}\) with tree shape, sequence length, model and rollout budget all fixed.
Cagnetta et al., Phys. Rev. X, 2024.
SECTION 02
exact — matches the suffix of the original sequence the prefix was cut from, token by token. A canonical-answer verifier: a synonym counts as wrong.
parse — any grammatical continuation. Score: the fraction of tree levels at which prefix\(+\)suffix parses. A validity verifier: a synonym counts as right. “Root-valid” means grammatical all the way up.
Start every mechanism from the same place
A separate model for each \(m\), pretrained until its loss plateaus. Every mechanism below starts from that checkpoint, so the question is always what reward does to a given basis.
Two kinds of result
Bounds: the best any optimizer could do, computed exactly on the known tree. Attainment: what each mechanism actually captures, and at what cost.
Why end at expert iteration
Sample, keep what the verifier passes, fine-tune on it. Every policy-gradient method is a noisy version of that loop. If the clean version cannot teach the model a new rule, no noisier variant will.
SECTION 03
Even at \(m{=}1\) the ceiling is 0.842, not 1: two root rules can share a left half, so the prefix does not always pin the root.
Computed exactly on the known tree, so no optimizer can beat it. Past \(m{=}1\) the prefix barely helps: 32 tokens in, the suffix is nearly independent of it, and the canonical-answer reward has nothing left to teach.
What remains is sharpening: always emitting the most likely answer instead of sampling one. That lifts accuracy without any new knowledge (\(0.19\to0.28\) at \(m{=}2\)).
The validity verifier runs the other way
More synonyms means more grammatical strings, so its reward gets denser on the same knob. How hard a domain is for RL depends on the verifier you pair it with.
Same 2.7M-parameter model at every \(m\), trained to plateau. Colour \(=\) how much of what is learnable at that level the model learned, read off how far its next-token loss sits above the best achievable (the Bayes floor). That gap, in nats, is the “excess” quoted from here on.
SECTION 04
At every \(m\) the model ends 10–14 nats above the floor, worse than uniform guessing, and its policy has collapsed to one answer per prompt (greedy \(\equiv\) sampled).
Diagnosis
The reward was climbing when it crashed. The signal was there; the optimizer destroyed the model that was reading it.
Add a KL penalty toward the frozen pretrained policy and the destruction is cured: \(0.05\)–\(0.8\) nats above floor, no collapse. What is left, as sampled-policy accuracy:
| \(m\) | exact: pretrained \(\to\) RL | calibrated sampler* | parse: pretrained \(\to\) RL | nats above floor |
|---|---|---|---|---|
| 1 | 0.836 → 0.841 | 0.842 (ceiling) | 0.977 → 0.758 | 2.93 |
| 2 | 0.191 → 0.190 | 0.191 | 0.980 → 0.993 | 0.29 |
| 3 | 0.157 → 0.157 | 0.160 | 0.919 → 0.925 | 0.07 |
| 4 | 0.137 → 0.136 | 0.136 | 0.817 → 0.834 | 0.42 |
| 6 | 0.130 → 0.133 | 0.130 | 0.731 → 0.743 | 0.79 |
*the accuracy a model gets by sampling from the true answer distribution rather than picking its single best guess.
Canonical: zero gain at every \(m\ge2\). The policy sits exactly where a calibrated sampler would, as the bound says it must. At \(m{=}1\) it now holds the ceiling.
Validity: a \(+0.006\) to \(+0.017\) polish at \(m\ge2\).
What the anchor buys
The anchor stops the damage. It does not start any learning.
Root-level validity: share of sampled generations that parse all the way to the root. Validity verifier, final round.
So
Whatever selecting and fine-tuning on winners can do, this is the arm that does the most of it.
The dissociation
Root validity doubled, and root validity requires the deep rules. Yet no deep rule was learned: probability mass moved between sequences the basis could already produce (\(\eta^2\) unchanged; \(m{=}4\) shows the same).
Canonical verifier: exact-match gain from six rounds of EI
Validity verifier: share of pretrained knowledge erased by six rounds of EI
erased \(=\) 1 \(-\) (learned after EI) \(/\) (learned before), with “learned” the pyramid’s colour scale read over the whole sequence.
Same mechanism, same budget
As synonymy rises, reward buys less on the one-answer verifier and erases more of the pretrained distribution on the many-answer one. The model gets more fried as the task gets more sophisticated, even under the best-behaved mechanism in the class.
Across 3 mechanisms \(\times\) 5 \(m\) \(\times\) 2 verifiers, deep-rule knowledge stayed where pretraining left it. Everything else varied:
In language terms
Pass rates go up, the model still does not know the rules, and the distribution pays for it. Only a known generating process lets you tell those three apart.
Not all language tasks are created equal. Code and math sit at the left of this dial, creative writing and original thinking at the right, and language-model RL works on the former and not on the latter.
The one channel that could teach a missing rule is the one that looks like pretraining: fine-tuning on selected samples is more data. At this budget it taught none, while the reward doubled.
Takeaway
Continual learning that expands what a model can do needs a mechanism that moves the basis boundary. Vanilla policy reward is, structurally, not it. The natural objection is that RL on real language behaves differently. Said explicitly, that objection is a bet: for RL to carry continual learning, it would have to gain capabilities on language that it does not gain on this compositional grammar.