Caveman is a Claude Code plugin with 97,000 GitHub stars — 86,000 when I ran these benchmarks. It saves you tokens by making the AI drop the filler and grunt. Fewer words out, smaller bill.
I tried it. The AI really did get terser on five of the seven jobs I ran; on the other two it got wordier. My bill didn't budge.
Not all tokens cost the same
You pay for four kinds of token, at very different prices. On Sonnet 4.5, what these runs were billed at, per million:
- output (the words the model writes back) — $15
- cache-writes (putting context into the cache the first time) — $3.75
- input (the text you send) — $3
- cache-reads (context it has already seen) — $0.30
So one word the model writes costs as much as fifty words it re-reads. (Sonnet 5 is cheaper across the board and in the same ratios — output is still 5× input and 50× a cache-read — so every number below holds there too.)
And it re-reads a lot. Every turn, the model goes back over the whole conversation: your files, the test output, everything said so far. Then it adds a few words at the end. Those few words are about 0.6% of the tokens in a session. The other 99% is re-reading.
Token economics
Ceiling, not a saving
If the model wrote nothing at all
0.6%
Output only — the rest reloads every turn, however terse the answer
OutputEverything else (mostly cache-read)
Now count it in dollars →
Few words, but each one fifty times dearer — so in money they come to about 20% of your bill (the arithmetic is in the appendix).
That 20% is a ceiling, not a result. It's the whole slice a skill can touch by making the model's words shorter. The other 80% is context: 65% re-reading it, 15% writing it into the cache the first time. No amount of terseness makes either of those smaller.
65% of a fifth is 13%
Take that ceiling and multiply it by how much shorter the answers actually get:
- Answers get 100% shorter — the model says nothing at all → bill drops 20%. The ceiling, and impossible: you'd get no answers.
- Answers get 65% shorter, which is what the README advertises (pinned) → 65% of 20% = 13% off the bill. The best you could hope for if the claim were true.
- Answers get 6% shorter, which is what I measured on average → 6% of 20% = 1% off the bill. What you actually get, and it's inside the noise.
20% is the ceiling. 13% is the best case if the label were true; 1% is what I actually measured. The label quotes 65%, which is none of the three.
Thirteen percent isn't nothing — I'd install a tool that reliably did that. But nothing I tested got near it. (On a subscription your "bill" is a quota whose weighting Anthropic doesn't publish — but it has to sit somewhere between raw token counts, where output is 0.6%, and API prices, where it's 20%. Both ends of that range say the same thing.)
What the runs showed
Seven coding jobs, from a one-line helper up to a multi-file refactor. Each one done ten times: five with the skill, five without. That's 70 runs, or 140 counting the second skill I checked in the appendix. Ten times each, because the same prompt costs a different amount every time you run it. Each run's real dollar cost was metered with vigiles, an open harness I built, and every run had to pass a correctness check — no "savings" from broken answers. You can run the whole thing yourself, on your own subscription.
The skill was unmistakably live: its SessionStart hook fires from the first message, and the answers came back telegraphic. Yet output shrank by about 6% on average, not 65%. And that average hides the real result: it cut output hard on two jobs (−31% and −28%), and made it longer on two others (+54%, +21%). The bill came out 1% cheaper, less than the same task swings between two runs with no skill at all. The honest reading of both numbers is no change I could see, not a 1% saving off a 6% trim.
Could seven jobs miss a real saving? Not one the size of the label; that's excluded. One inside the ceiling, easily — which is why the argument never rested on the measurement in the first place. The ceiling is arithmetic (prices times shares) and you can redo it without my benchmark: 65% of a fifth is 13%, whatever my runs did. What the runs answer is the thing arithmetic can't, whether the skill gets anywhere near its own ceiling in practice. It doesn't.
I'm not the first to check the label. Max Taylor benchmarked caveman against the two words "be brief" and found it didn't beat that boring default on quality or on tokens. JetBrains ran one study over 86 real coding tasks with the skill forced on, which is the best case rather than a typical one. Output fell 8.5% against the advertised 65%. On the bill they expected about 10% cheaper per task, and in their words "routinely erased by single-trial variance" — but their own pooled totals came out 11.6% more expensive ($40.60 against $36.39), one outlier trial flipping the sign. Per task they expect a saving; pooled they got a penalty; neither is anywhere near 65%. That study also shows how badly single runs lie: their ten-task smoke test, run once, said −29.5%; the same ten tasks at three runs each said −6.7%. Their headline 86-task number is itself a single run per task. Mine ran five.
Where the 65% came from
Not a lie — a measurement of the wrong thing. The headline 65% was self-measured on 10 one-shot prompts (their own range: 22–87%). One prompt, one reply, no files, no tools, no re-reading — the one setting where the reply basically is the whole session, with no re-reading for it to get lost in. Trim it there and you trim almost the whole token count.
Real coding is the opposite, and the number shrinks twice. First the slice: the reply that was the whole session is now a fifth of the bill. Then the cut itself. Code and structured edits compress far less than the rambling "helpful assistant" prose their baseline measured against, so the same skill trims about 6% on average, and on two of the seven tasks it grew output instead. The label was measured in the one world where it's big, then quoted for the world where it's tiny.
▶The full numbers (per-task, both skills, p-values)
Where the 20% comes from. Output is ~0.6% of the tokens at ~50× the cache-read price. On a napkin:
volume price cost
writes 0.6% × 50 = 30
re-reads 99% × 1 = 99
output's share of the bill: 30 / 129 ≈ 23%The napkin says output is ~23% of the bill; measured against the recorded dollar cost of my runs it's 20.5%, a touch lower because real sessions also pay for input and for writing to the cache. The body rounds that to 20%.
Two skills here, not one. Caveman is the 65% headline the article is about; token-efficient is the next most-starred skill in the same category, advertising a 63% cut the same way — I ran it through the identical harness to check whether falling this far short of the ceiling is a caveman thing or a category thing. It's the category.
Seven tasks × five trials × two arms = 70 runs per skill, 140 across both, on Sonnet (~$10
API-equivalent, $0 on my subscription). Output change per task — negative means the skill cut output, positive means it
grew — with a Welch p-value, a check that doesn't assume both arms are equally noisy (they never
are). Read the p as "the odds of seeing a gap this big if the skill did nothing"; * marks
p<.05. The two Caveman cuts (.002, .006) survive a Bonferroni correction, which just means:
testing seven things at once makes a fluke more likely, and these two still hold up after allowing
for that.
| Task | Caveman outΔ | p | token-efficient outΔ | p |
|---|---|---|---|---|
| slugify | +54% | .27 | −2% | .78 |
| debounce | −28% | .22 | −7% | .52 |
| bugfix-offbyone | −31% | .002* | −6% | .87 |
| bigO | −18% | .27 | +6% | .79 |
| regex-email | −28% | .006* | +127% | .037* |
| review-doc | −8% | .80 | +54% | .46 |
| refactor-suite | +21% | .65 | +33% | .19 |
Pooled dollars across every run: Caveman −1%, token-efficient +10%. Both are smaller than this study can resolve, but they differ in character: token-efficient got more expensive on 5 of the 7 tasks (regex-email +49%, review-doc +21%, refactor +9%, slugify +7%, bigO +6%; only debounce −13% and the bugfix −5% got cheaper), while caveman's −1% is noise around zero. Averaged per task Caveman's output cut is 5.6% — the body's "about 6%"; pooled it is 9.1%. Either way a ~20% slice turns that into 1–2% of the bill, and the pooled dollars came out −1%. Output is ~20% of the dollar cost in both arms, and every answer stayed correct: zero correctness regressions across all 140 runs.
The skill isn't free. Its always-loaded prompt is itself context: across every run the caveman arm carried 9% more cached tokens than baseline, and on the shortest task 71% more — which is why the cheapest task came out 66% dearer. That's a systematic penalty on short sessions, not only noise.
The spread. Per task, the dollar change swings from 16% cheaper (the bugfix, p=.03) to 66% more expensive (slugify — the cheapest task, where a little run-to-run noise looks enormous as a percentage). With five trials a task, the 95% CI on the pooled bill change runs from 14% cheaper to 12% dearer, so this design can't cleanly resolve a cost effect below about 13%. A 13% saving — the label's own best case — would have sat right at the edge of what these runs could see. It doesn't need to: the precision comes from the ~20% structural ceiling, not the sample size — you don't need a big study to rule out a big saving when the ceiling on the saving is itself small.
One aside: the API's per-minute rate limits are a separate axis from cost — cache-reads mostly don't count toward the input-rate limit and output has its own bucket. If you're pinned against the output-per-minute limit specifically, trimming output does buy throughput; that's a narrow case, and it isn't your monthly bill or quota.
The correctness gate is a structural check, not a quality judgement: it bounds information loss, it doesn't prove the answers were as good.
Per-arm measurements from all 140 runs (means, σ, n): the JSON, and the
/caveman-compress test. Method + harness:
vigiles.
Where the savings actually are
Caveman's backup trick — compressing your instructions file — is a rounding error too: I shrank one by 68% and the bill didn't care — a few-hundred-token file next to a ~100,000-token session.
Input compression · /caveman-compress
file −68%, bill −14% (n.s.)
Your session bill−14%noise (p=.36)
Cache-read tokens−2%noise (p=.87)
It compressed the conventions file 68%, beating its own 46% claim. The bill moved 14%, but that's inside the run-to-run noise: a 500-token file is a rounding error next to a 100,000-token session.
Want a smaller bill, or more runway before your limit? Don't squeeze the words — squeeze what reloads every turn:
- A 10k-line test log or a sprawling
git diff, pulled in once and then silently re-read every turn after. That pile is what grows your bill. - A conversation nobody trims. Hand work to subagents so the main thread stays small.
/compactcuts the big cheap pile directly, which is almost all your cost.
I haven't benchmarked any of these, so: leads, not promises. But that's where the money is.
Before you install the next one
The question I'd ask now is which tokens it cuts, and whether anyone measured the actual cost on a real session rather than the token count. "It makes the model write less, measured on one prompt" answers both, and not in the tool's favour.