When OpenAI released gpt-6-astra, I noticed some strange output in the model-written subagent prompts. Codex encrypts these, but I run a custom harness that doesn’t. The prompts contained long runs of words without any spacing:
FreezeincludesyournewdiagnosticfilewriteswhilebenchmarkThe marketing surrounding Astra touted its token efficiency, it seems they sacrificed readability in pursuit of that.
Was that a good price to pay? I want you to guess how much less efficient it is to make this a little bit more readable, just by adding some spaces:
Freeze includes your new diagnostic file writes while benchmarkWrite your answer in an envelope!
This article assumes a familiarity regarding “roughly how LLMs work”, requiring knowledge of concepts such as “LLMs ingest and output tokens”. The
o200ktokeniser is used for all visualisations.
This article has been updated due to exciting news…
Efficiency
The first rule of optimisation is to “optimise for the real objective”. Proxy objectives are a monkey’s paw, so let’s define what we want.
We want something useful, we want it for low cost (of time and money, but let’s assume throughput is fixed). LLMs are billed per token, so cost can be approximated by that token count.
An LLM does its machinations in its forward passes. The tokens in the context are just the state it maintains for the next pass. You can do more compute in each forward pass, but ultimately the information density of the tokens entering context is a limiting factor.
The vast majority of tokens are ‘internal reasoning state’, unseen by the user. Those tokens should be as information-dense as possible. It all comes down to compute proportional to entropy.
Tokenisers
Astra uses the o200k tokeniser, which is in the ‘Byte-Pair Encoding’ family. They require a pre-tokeniser to break up text into words before the actual frequency based merging operation.
Each caret you see is a 'word' boundaryNo token can exceed the interval defined by the pre-tokeniser, so it’s a limiting factor for how many bytes you can stuff into one token. These forced boundary points will be denoted by a caret ^.
The pre-tokeniser is commonly defined by a regex. I want to show you the absolute beauty that’s used in the o200k pre-tokeniser:
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+(?i:'s|'t|'re|'ve|'m|'ll|'d)?|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+dusts off the syntax highlighter…
[^
\r\n\p{L}\p{N}
]?
[
\p{Lu}
\p{Lt}
\p{Lm}
\p{Lo}
\p{M}
]*
[
\p{Ll}
\p{Lm}
\p{Lo}
\p{M}
]+
(?i:
's|'t|'re|'ve|
'm|'ll|'d
)?
|
[^
\r\n\p{L}\p{N}
]?
[
\p{Lu}
\p{Lt}
\p{Lm}
\p{Lo}
\p{M}
]+
[
\p{Ll}
\p{Lm}
\p{Lo}
\p{M}
]*
(?i:
's|'t|'re|'ve|
'm|'ll|'d
)?
|
\p{N}{1,3}
|
?
[^
\s\p{L}\p{N}
]+
[\r\n/]*
|
\s*
[\r\n]+
|
\s+
(?!\S)
|
\s+Yikes. Well you know what they say about eating the elephant, one bounded interval at a time.
First off, the Optional Prefix. It matches one character that isn’t a bunch of things:
It’s kind of poetic that we’re defining something by what it’s not, LLMs do love that…
The Optional Prefix therefore can contain:
- nothing (it’s optional!)
- a space or tab:
- a symbol:
~ ! @ # $ % ^ & * ( ) - an emoji:
- or technically some other esoteric unicode stuff that isn’t relevant.
This means !important can become one token (one leading symbol, run of lowercase letters) but !!important cannot. It will instead be bounded by a run of symbols, then a run of letters.
Next are the Uppercase then Lowercase sections. This is the part that contains the actual letters. The doubling up of the Prefix, Uppercase, Lowercase sections in the regex is to enforce an ordering of upper then lowercase letters without matching on empty segments. We’ll just run through one of them.
[
\p{Lu}
\p{Lt}
\p{Lm}
\p{Lo}
\p{M}
]*
[
\p{Ll}
\p{Lm}
\p{Lo}
\p{M}
]+This will match on:
- Uppercase then Lowercase:
Hello - all Lowercase:
world
And the alternative Uppercase, Optional Lowercase will match on:
- all Uppercase:
HELLO
Since the regex match in both cases is Uppercase then Lowercase, the opposite Lowercase to Uppercase transition will always result in a boundary at that case transition.
helloWORLD can never become one token, no matter how often it happens in the training data.
Finally the Optional Contractions:
(?i:
's|
't|
're|
've|
'm|
'll|
'd
)?This just allows common contractions like you're to potentially stay as one token.
Overall, the pre-tokeniser forces this behaviour:
lowerUPPERwill never become one token.longrunsofsamecasecan become one token.- a leading
spacecan be ‘absorbed’ into a token. mixing_symbols__and___textonly absorbs a lone leading symbol, eg!important
There’s nothing in the pre-tokeniser stopping this from becoming one token!
FreezeincludesyournewdiagnosticfilewriteswhilebenchmarkAs long as the tokeniser’s training data contained this kind of thing, this could very well be more efficient!
Whitespace
Let’s actually count the tokens.
Freeze includes your new diagn ostic file writes while benchmark
10 tokens·5.5 bytes per token
Almost one token per word, so that’s pretty good, right?
How much worse is the added whitespace?
Take out your envelopes
Freeze ·includes ·your ·new ·diagnostic ·file ·writes ·while ·benchmark
9 tokens·7 bytes per token
Oh it’s better. Another example:
This co herent l ifecycle pol ic yp re ferred over fin item ode heur istic
16 tokens·3.69 bytes per token
This ·coherent ·lifecycle ·policy ·preferred ·over ·finite ·mode ·heuristic
9 tokens·7.44 bytes per token
Oh that’s much better. The no-whitespace version breaks the word-token alignment, creating inefficient fragments.
Across a sample of 2,217 inter-agent messages in my sessions, totalling 181,749 tokens, the more readable version had:
- 6.88% more characters and
- 4.77% fewer tokens.
In hindsight, that’s not too surprising. Most text has spaces between words, and the tokeniser was trained on a lot of ‘normal text’. The pre-tokeniser explicitly includes leading spaces so that they’re absorbed into words.
Interestingly, Astra doesn’t always delete whitespace. Here it kept the space before ‘deliberately’:
extension line ·deliberately not added until RED
7 tokens·6 bytes per token
Notice that removing the space affects the token before the space:
extension lined el iber ately not added until RED
9 tokens·4.56 bytes per token
To remove that space, the model would need to emit lined instead of line , deciding to elide the space before it reached the space-containing token.
We know models can do this kind of ‘thinking ahead’, it’s how they’re able to write rhymes. However, Astra seems to have simply learned a cheap, per token decision.
Across the sample, Astra dropped about 70% of spaces removable just by picking the next token without a leading space, whereas it dropped less than 30% of the ones that required that kind of planning.
But they optimised for token efficiency!
Did they optimise for token efficiency?
The first rule of optimisation is “optimise for the real objective”.
If you read the OpenAI online tokeniser playground notes below the text area:
A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text.
If you dig around the codex repo, there are multiple copies of a function like approx_token_count():
const APPROX_BYTES_PER_TOKEN: usize = 4;
pub fn approx_token_count(text: &str) -> usize {
let len = text.len();
len.saturating_add(APPROX_BYTES_PER_TOKEN.saturating_sub(1)) / APPROX_BYTES_PER_TOKEN
}I am pretty certain that this estimate has replaced an actual token count within some part of OpenAI’s RL pipeline.
As a result, demonstrably for ‘inter-agent’ communications, and likely for internal reasoning in general, the model has been optimised to produce fewer bytes instead of fewer tokens. That proxy objective is resulting in exactly the opposite of the effect desired.
Bonus Update - Taking a peek at some internal reasoning
It’s the 21st of October. OpenAI has been publishing Misalignment Reports, and inside them are snippets of internal reasoning. They are missing a lot of whitespace!
Need ·ask notifications ·if send no -- ·since ·no Slack ·with write ·no _notify true ·prevents alerts but ·disabling ? ·parent ·shouldn't set ·no _notify true ·since ·policy ·trigger user no ·messages ·requires not pro active ·tool . ·Artifact actions central autom atics ·okay ·scope ? ·Better ·flag ·sink ·emits ·no ·direct ·notifications ·but ·separate central watch may notify ·if P 0 ·doesn't need ·ourselves ·send . ·User explicit nom essages ·originally ·should keep n ore c ipient . ·Existing ·default ·ping empty ·enough central not ifier alerts ? ·no _notify ·only ·way ·prevents ; ·but ·disabling required alerts unsafe . ·Parent ·aware policy . ·Don't strong adv ise ·no _notify .
112 tokens·5.19 bytes per token
Looking through these snippets, there are heaps of spaceless runs that can never become one token:
noSlack- a lowercase to uppercase transition.doesn'tneed-'tneedisn’t a contraction in the regex.
If we count their ‘added spaces’ version:
Need ·ask ·notifications ·if ·send ·no -- ·since ·no ·Slack ·with ·write ·no _notify ·true ·prevents ·alerts ·but ·disabling ? ·parent ·shouldn't ·set ·no _notify ·true ·since ·policy ·trigger ·user ·no ·messages ·requires ·not ·proactive ·tool . ·Artifact ·actions ·central ·autom atics ·okay ·scope ? ·Better ·flag ·sink ·emits ·no ·direct ·notifications ·but ·separate ·central ·watch ·may ·notify ·if ·P 0 ·doesn't ·need ·ourselves ·send . ·User ·explicit ·no ·messages ·originally ·should ·keep ·no ·recipient . ·Existing ·default ·ping ·empty ·enough ·central ·notifier ·alerts ? ·no _notify ·only ·way ·prevents ; ·but ·disabling ·required ·alerts ·unsafe . ·Parent ·aware ·policy . ·Don't ·strong ·advise ·no _notify .
107 tokens·5.79 bytes per token
That’s 4.46% fewer tokens. Across all the excerpts on that page combined, we can save 5.96% of the reasoning tokens!
I wonder if there is a causal link between the agent misbehaviour and this byte-count pressure?
What is more token efficient?
If you’re willing to trade readability for fewer tokens, removing a space can help in some particular cases. Numbers are in their own regex section and don’t have ‘space prefix’ versions.
A ·cat ·has · 9 ·lives
6 tokens·2.83 bytes per token
As a result, A cat has9 lives with no space between has and 9 will be one token more efficient and slightly less readable.
A ·cat ·has 9 ·lives
5 tokens·3.2 bytes per token
The various combinations of spellings and case also create different tokenisations:
lowerTitleUPPER
US, leading space ·authorized ·Authorized ·AUTH ORIZED US, elided space authorized Authorized AUTHORIZED British, leading space ·authorised ·Author ised ·AUTHOR I SED British, elided space author ised Author ised AUTH ORI SED
The model picked this sequence:
Main AUTHORIZED post
3 tokens·6 bytes per token
Which is fewer tokens than:
Main ·AUTH ORIZED ·post
4 tokens·5 bytes per token
But avoiding upper case and keeping the spaces would have worked just as well:
Main ·authorized ·post
3 tokens·6.67 bytes per token
You could also:
- make tokens that are ‘forced’ by a grammar essentially free
- get rid of line numbers or per-line-paths from file reads
- get rid of line and character number based MCP tooling
- allow Lark grammars and text calls for MCP tooling
- give the model the ability to copy and paste
- write a ‘token efficient’ minifier, have the model ingest and output that, then format it as it’s written to a file
- make AST based editing standard for coding agents
Repairing an RL run with speculative decoding
It would be a one line fix to use the real tokeniser instead of the estimate for any future training.
-approx_token_count(text)
+actual_token_count(text)But what about the expensive RL runs that are already contaminated?
If we can recover the well spaced text, we can do a little bit of training to move the model back on track. We can do that cheaply by taking inspiration from speculative decoding! First we split our text by whitespace and find the characteristic long runs:
FreezeincludesyournewdiagnosticfilewriteswhilebenchmarkWe then perform parallel passes of ‘prefill’ pf-over the text prefixes and extract the log probabilities of every possible continuation token. There are several choices for what prefix context can be used before the long run, but it worked well enough with nothing that I stuck with that.
I’m using
gpt-oss-20bhere because it’s tiny, uses theo200ktokeniser, and is smart enough to know where spaces go between words.
We maintain a baseline set of tokens, starting with the original model’s version. Spaces can go between any pair of characters. We compare each candidate’s first token against the overlapping tokens in the baseline. Any candidates that beat the baseline continue. We can cache prefixes and do tree attention for some efficiency gains (eg the in is shared between multiple suffixes). For this example there are 54 candidates, this process culls that to 26.
Freeze in ·cl udes ✓ Freeze in clud ·es ✓ Freeze ·includes ✓ Freeze includes ·your ✓ Freeze includes your ·new ✓ Freeze includes you ·r ✓ Freeze includes your new - Freeze include ·sy our × Freeze includes your ne ·w × Freeze includes yo ·urn ew ×
Some winning candidates change the subsequent tokenisation of the characters after them (like in ), so we have to run another prefill pass to collect their log probs.
A high probability first token doesn’t guarantee a winning sequence. in is more likely than ·includes , but when we add in the clud ·es continuation, it loses.
Some of the winning candidates are a single token (like ·includes ) and can be compared via the cached log probability.
Freeze in ·cl udes × Freeze in clud ·es × Freeze ·includes → Freeze includes ·your → Freeze includes your ·new → Freeze includes you ·r × Freeze includes your new -
We take the winning ‘edits’ to form the next baseline , and rerun from the first changed token. Over 4 rounds of this, 8 total prefill passes, we recover the correct spacing.
8 prefill passes for 9 words is pretty bad. It demonstrates just how out of distribution this text is compared to what the model expects.
We can make our ‘first draft’ more likely with some cheap heuristics. A dictionary ranked by word frequency lets us ‘fit’ the sequence against actual space-prefixed words with a bit of dynamic programming. This gets us 90% of the way by ‘spaces count’, except it has trouble with compound words (in take vs intake) and plurals (events can vs event scan).
Much better. In this case, the dictionary draft was 100% correct, so we check the continuations and are done in 2 passes instead of 8. That’s the trick with speculative decoding, if you predict correctly, it’s faster!
OpenAI runs actual speculative decoding to increase the throughput of their models. Even if that spec decode model is distilled from the main one, I bet there are second-order efficiencies lost by this being so out of distribution.
Now that we have the correct token sequence, we can do the minimum amount of post-training to fix this. Anchor the token distributions we’re not touching with a KL penalty, then do the smallest push to flip the tokens at each prefix.
message ":" Freeze includes your new message ":" Freeze ·includes ·your ·new
Back on track.
untilnexttimeIf you would like further reading on this topic, I recommend:
- On the Biology of a Large Language Model - An excellent piece on mechanistic interpretability in LLMs.