Do Skills Improve Coding Agent Accuracy?

Agent skills were originally designed to allow developers to teach a general AI agent a specific process or domain expert’s workflow without bloating the agent's prompt or context window. Before agent skills, managing AI behavior was a messy process of swapping and chaining system prompts.

Almost immediately, developers such as Jesse Vincent ¹ were developing skills to improve Claude Code and other harnesses' ability to handle general engineering tasks. Jesse's Superpowers ² claimed to impose a professional methodology automatically; Get Shit Done (now Git Ship Done or simply GSD) ³ would turn an idea into durable specifications and phases; Oh My ClaudeCode ⁴ then promised to provide “zero learning curve” orchestration. Andrej Karpathy's tweets ⁵ about his coding workflow got quickly turned into another skill ⁶ and Addy Osmani released "Agent Skills" ⁷, production-grade engineering skills for AI coding agents.

None of them originally shipped with evidence, other than some demonstrations, that they actually improved end-to-end software-engineering performance. Superpowers came closest, but it tested workflow compliance, not whether the same model solved more coding tasks. And with the AI models and coding harnesses constantly improving, it was an open question whether any gains would survive the next model release.

So I decided to put these skills to the test using similar benchmarks to the ones that the frontier labs use when they release new models: SWE-bench Pro ⁸ and SlopCodeBench ⁹. The same arms are re-run every month. Skill packs ship updates, models drift underneath them, and a number published once decays quietly. Each run below is a dated snapshot with its own config; the methodology is shared, so the months are directly comparable.

Results Combined

Both runs side by side, one section per suite. Every delta is measured against that month's own baseline, which is the only honest way to compare across runs.

Codex 5.5 changed underneath us, and nothing in its version string says so. Same harness (Codex CLI v0.136.0), same model name (gpt-5.5, reasoning effort high), one month apart. Across that month the skill-free baseline gained +2.76 points on SWE-bench Pro and +1.7 Strict on SlopCodeBench, and its verbosity roughly tripled, from 0.311 to 0.827. Those are improvements OpenAI made server side, and there is no revision number anywhere for a reader, or a benchmark, to pin them to. Every figure below is a measurement of a moving target, which is why each run is dated and compared only against its own contemporaneous baseline.

SWE-bench Pro

Resolve %, by run. The right-hand column is each arm's month-over-month change; the bracketed figure is its edge over that month's baseline.

Toggle the runs to see it directly: the entire July cloud sits above June's, at roughly the same spread of cost. That vertical shift is the drift.

The single most important number on this page is the baseline's +2.76, achieved with the same harness version and the same model string. The model got better at these tasks in one month by more than any skill pack's advantage over it in either run. Skill effects here are small quantities riding on a much larger moving one, which is the entire argument for re-running rather than citing a number from a blog post.

Only two rows are clean drift measurements. Baseline runs no skill at all, and Karpathy Skills is the one pack that shipped nothing between the runs (same commit, 2c60614). Git Ship Done, Oh My ClaudeCode and Superpowers all moved commits, so their month-over-month figures mix model drift with a pack update and cannot be read as either one alone.

SlopCodeBench

This suite separates two things SWE-bench Pro blends together: how many checkpoints an arm solves outright (Strict), and how much previously-working code it breaks as the codebase accretes (Erosion, lower is better). Comparing both across the two runs characterises what actually changed in the model far more precisely than a single resolve rate can.

Capability moved, discipline did not, and verbosity roughly tripled. Every arm's Strict rate rose (+0.5 to +3.1) while every arm's Erosion moved by at most ±0.02: the model got materially better at solving new checkpoints and no better at all at not breaking what it had already written. Verbosity is the third axis, and it moved hardest. Every arm went from roughly 0.26–0.31 to 0.83–0.92, the skill-free baseline included, under the same harness version and the same model string. Codex 5.5 simply became a much wordier model in the space of a month.

The per-checkpoint curves say the same thing in a different way. Toggle the runs: the two bands sit almost on top of each other and fall at the same rate, from roughly 75–81% at checkpoint 1 to 33–50% by checkpoint 8. July starts marginally higher and ends marginally lower, a slightly steeper slide rather than a flatter one. Hover any line to identify the arm.

Two cautions when reading it. The problem count shrinks as checkpoints advance (36 down to single digits), so the right-hand side is much noisier than the left; the convergence at cp7–cp8 is partly an artefact of having few problems left. And no arm separates from the baseline anywhere along the curve, in either run, which is the same null result June reported.

That is pointed for skill packs specifically, because regression discipline is exactly what most of them claim to add, and it is the axis the model did not move on. Erosion also stays stubbornly arm-specific across both months (Superpowers ~0.46, Karpathy ~0.58), suggesting it is a property of the workflow rather than the model.

Karpathy is an accidental control arm, and a useful one. It ran on identical code in both months, not because we pinned an old commit, but because the repo has not changed since April. Its movement should therefore equal the baseline's if nothing but the model changed, and it does, almost exactly: +1.8 against baseline's +1.7 here, and +3.44 against +2.76 on SWE-bench Pro. Two independent arms agreeing on the size of the drift is the strongest evidence on this page that the June→July shift is real and not an artefact of one measurement.

Take that agreement as the yardstick and the pack updates can be scored against it:

• Superpowers v6 genuinely improved. +3.1 Strict against a ~+1.75 drift baseline is roughly double what the model alone delivered.
• Git Ship Done (+1.2) and Oh My ClaudeCode (+0.5) came in under drift, on both benchmarks. GSD managed only +0.97 on SWE-bench Pro where standing still was worth ~+2.8. Their July updates look net negative: they gave back some of what the model handed them.

Stated carefully, because the noise is real: SWE-bench Pro is single-seed with sub-1pp gaps, and SlopCodeBench spreads run ±0.5–2.3. These are directional readings, not settled ones, which is the argument for the next run rather than a conclusion from this one.

The verbosity ordering also inverted, which is worth separating from the size of the jump. In June every skill pack came in below the baseline (0.258 to 0.300 against 0.311): the packs were restraining output. In July every one sits above it (0.895 to 0.915 against 0.827). So the base model did not just get wordier, it got wordier in a way the packs now add to rather than damp down. For a pack sold on imposing discipline, that is the wrong direction on the one axis that moved most.

July 2026

This run exists to answer two questions the June run couldn't: how much of the June result was skill quality versus model drift in the underlying model, and whether the packs' own updates (notably Superpowers v6) change the ranking.

Both answers turned out to be uncomfortable ones.

SWE-bench Pro

729 instances common to every arm, single seed per arm.

Every gap here is under two points on a single seed, so treat the ordering as indicative rather than settled; the sign of each delta is the interesting part.

June's headline result did not survive. In June every skill collection beat baseline; in July two are below it. Git Ship Done went from +1.64 to −0.14 without getting worse in absolute terms. It scored higher than in June (54.45% → 55.42%) and still lost its edge, because the baseline improved faster underneath it. Its pack also shipped an update between the runs, so the loss is either drift outrunning it or a regression in the update; neither reading flatters the pack. Agent Skills, new to the cohort this month, lands at −1.10.

Karpathy Skills is now the efficiency standout: it matches the baseline's token spend and cost per problem exactly (1.37M, $0.37) while resolving ~1 point more. Every other pack buys its edge with 1.5–2× the tokens, and the two most expensive arms are the two that now lose to baseline.

SlopCodeBench

All 36 problems, n ≤ 3 seeds per arm.

Superpowers v6 is the first skill pack in either run to beat baseline on Strict (14.5 vs 13.9), reversing June's clean sweep in the other direction. Treat it gently: its ±2.3 spread is wider than its 0.6-point margin, so this is suggestive, not established. The win is narrow in a second sense too, since v6 simultaneously posts the worst Core (62.6) and Partial (40.7) of any arm. It solves more checkpoints outright while breaking more of what it had already built.

The baseline moved here too (Strict 12.2 → 13.9), so the same drift story applies, and baseline still leads on Iso, Core and Partial.

Every pack is also more verbose than baseline (0.83 vs 0.90–0.92), which is the cost story restated: the packs spend more words to land in roughly the same place. Karpathy is the exception that proves the rule: the only arm cheaper than baseline per checkpoint ($1.25 vs $1.26) while still finishing above the two most expensive packs.

Verdict

Measured against a contemporaneous baseline, the public workflow skills mostly wash. Which one "wins" depends on the benchmark and the metric more than on the skill.

• Superpowers-v6: the only pack positive on both (Pro +1.51, SlopCodeBench Strict leader). Caveat above: that Strict lead comes with the worst Core and Partial, on wide noise.
• Oh My ClaudeCode: Pro leader (+1.65) but 1.8 below baseline on SlopCodeBench Strict. Benchmark-dependent.
• Karpathy Skills: a cheap positive on Pro (+0.96 at baseline cost), negative on SlopCodeBench.
• Git Ship Done: neutral-to-negative on both, at the highest token spend on Pro.
• Agent Skills: negative on both.

The June answer to the title question was "sometimes, and it depends on the task." One month later the same experiment says something sharper: it depends on the task, the metric, and the month.

What changed since June

• The model, under a fixed version string. Same harness (Codex CLI v0.136.0), same model name (gpt-5.5, high reasoning). The baseline arm runs no skill at all and still rose 52.80% → 55.56% on Pro and 12.2 → 13.9 Strict on SlopCodeBench, while its verbosity went 0.311 → 0.827. That is silent server-side drift, and it is the cleanest measurement on this page.
• Three of the five packs also shipped updates: Git Ship Done (de73ad92 → 4fc89497), Oh My ClaudeCode (a1720433 → 41a4c0f7) and Superpowers (v5.1.0 → v6). Their month-over-month numbers therefore conflate drift with a pack change.
• Karpathy Skills did not (2c60614 in both runs), which is what makes it a usable control against the baseline.
• Agent Skills joined the cohort this month, so it has no June comparison.
• Set size: 729 instances common to all arms, against June's 731.
• New SlopCodeBench metrics (Verbosity, cost/checkpoint detail), which is why the June table has no Verbosity column.

June 2026

731 problems on SWE-bench Pro (single run) and 36 problems / 196 checkpoints on SlopCodeBench (mean of 3 runs). Model: Codex 5.5.

SWE-bench Pro

Agent Skills is absent here: it entered the study with the July cohort, so it has no June measurement.

SWE-bench Pro contains long-horizon issues drawn from 11 actively maintained open-source repositories; a task may require substantial coordinated changes across several files, but the agent generally gets one issue and one final evaluation.

Note: While these benchmarks were running, Jesse Vincent released Superpowers v6, which is covered in the July run above.

Spending more per problem did not buy accuracy: Karpathy was the cheapest arm and still beat the baseline, while Git Ship Done was the most expensive without being the most accurate. Plot it in the combined SWE-bench Pro chart by toggling June on its own.

SlopCodeBench

Note the verbosity column: in June every skill pack was less verbose than the skill-free baseline (0.258 to 0.300 against 0.311). By July that ordering has inverted and the whole scale has roughly tripled; see Results Combined.

SlopCodeBench contains 36 synthetic, language-agnostic problems divided into 196 sequential checkpoints. The agent receives only an observable CLI or API contract, chooses its own architecture, and must keep modifying the code it previously wrote.

Every arm eroded together and none pulled ahead of the baseline. See the per-checkpoint curves, toggled to June alone.

Verdict

For SWE-bench Pro, all the skills provide an improvement over baseline. This effect disappears with SlopCodeBench with all strict scores worse than baseline (although GSD gets higher Iso and Core pass rates).

With the exception of Karpathy Skills, each collection of skills costs more to run than baseline.

On the evidence: sometimes, and it depends on the task.

• On SWE-bench Pro, yes, uniformly. Every skill collection helped.
• On SlopCodeBench, no: every skill collection actively hurt.

A caveat worth stating plainly: SlopCodeBench ran at n=3 seeds and the spread on several arms (±2.3) is wider than the gaps between them.

Potential explanations

Many of these frameworks contain procedures intended for navigating and modifying established codebases. Those procedures have limited value during the initial greenfield checkpoint of SlopCodeBench and may impose a context or orchestration cost. SWE-bench Pro's hard part is finding the right 20 lines in a 500k-line unfamiliar repo and making a surgical edit whereas on SCBench the agent wrote the code itself so it will usually already be in context. Additionally all these instructions, routing decisions and procedural constraints add to the context. On a difficult repository task, that additional structure can focus the model. On a small task, it can compete with the actual specification for attention.

In SWE-bench Pro, the repository normally provides existing tests, related test patterns and observable regressions. A TDD or systematic-debugging workflow can use that evidence to localize the issue and protect surrounding functionality. SlopCodeBench keeps its evaluator tests hidden. Agents can write tests only from the current external contract and examples. Skills push "reproduce the failure, write a failing test, then fix." but on SCBench there's no bug to reproduce.

Methodology

The methodology below is shared by every run above. When it changes, the change is noted in that month's section and the affected months are marked as not directly comparable.

Try it yourself in OrcaBot

There is a template for running SlopCodeBench within OrcaBot which acts as the benchmark orchestrator for you with a live browser showing the progress and results.

Model and harness

All runs used OpenAI Codex 5.5 (gpt-5.5, reasoning effort high) via the Codex CLI (v0.136.0) on a ChatGPT subscription. The agent runs inside each task's Docker container: the harness starts the container, docker execs Codex into it with the skill mounted, and extracts the resulting git diff. No agent logic runs on the host.

• SlopCodeBench: robdmac/slop-code-bench @ reproduce-public-skills
• SWE-bench Pro: robdmac/SWE-bench_Pro-os @ reproduce-public-skills

Skills: real, not distilled

Each skill's actual upstream repository was mounted read-only into the container at a fixed commit (below); the trigger prompt instructs the agent to read the repo's own entry file (e.g. OMC's AGENTS.md) and follow it. We did not paraphrase any skill into the prompt; trajectories confirm the agent read the real skill files (e.g. 730/731 OMC runs opened oh-my-claudecode/AGENTS.md and its rule files). Because Codex is a single agent, multi-agent frameworks (OMC, Superpowers) were applied as a single-agent sequential pass rather than orchestrated sub-agents; this is a faithful adaptation but may understate skills designed around native multi-agent tooling.

Pinned commits per run. Every pack except Karpathy shipped changes between the two runs, which is exactly why the baseline and Karpathy arms matter as controls (see Results Combined).

On the Karpathy pin. Both runs use the same commit not because we froze it, but because the repository has not changed: 2c60614 (2026-04-20) is still the latest commit upstream, with zero commits since. That is a property of the pack, not a choice we made, and it is what makes the arm a fair control rather than an artificially stale one; the pack was tested at its current state in both runs.

The contrast is stark. Checked on 2026-08-07, three of the other four packs have already moved past the commit used in the July run: Git Ship Done to a731a45 (that same day), Agent Skills to d2478bf, Superpowers to 44c9b2d. Only Oh My ClaudeCode still sits at its July commit. Any conclusion below about a pack, as opposed to the model, has a shelf life measured in weeks, which is the other half of the argument for re-running.

SWE-bench Pro Configuration

• Set: the public split, 11 repositories: 731 instances in June, and the 729 instances common to every arm in July.
• Generation: one attempt per instance (n = 1 seed), 30-minute cap.
• Evaluation: the official Docker-based evaluator, run locally. We patched one bug: the Docker SDK's 60-second client read-timeout silently drops output for test suites that run longer than 60s (common in Go/JS repos), mis-scoring them as failures; we set the client timeout to 3600s. Tests were scored serially.
• Resolve % = fraction of instances where all required tests pass (FAIL_TO_PASS ∪ PASS_TO_PASS). Partial % = mean fraction of required tests passing per instance (partial credit).

SlopCodeBench Configuration

• Set: 36 problems / 196 checkpoints. Each problem is greenfield at checkpoint 1, then iteratively refined; regression tests from prior checkpoints accumulate.
• Generation: n = 3 seeds per problem; results are the mean over seeds, ± the run-to-run population standard deviation of the whole-benchmark rate.
• Evaluation: serial (SCBENCH_PYTEST_WORKERS=1) to avoid xdist-order flakiness on parallel-sensitive problems.
• Strict = checkpoint solved iff every required test passes; Iso = solved ignoring prior-checkpoint regressions; Core = core tests only; Partial = % of problems with ≥1 strict checkpoint; Erosion = degradation of earlier checkpoints as code accretes (lower is better).

References