Opus vs GLM-5.2 in a coding-agent pipeline — paired-run findings

GLM Tries, Opus Triages: Behavioral Differences in Research-to-Code Agents

A controlled comparison across 19 paired runs spanning 19 repository forks — 38 individual workflow executions total — running an identical paper-implementation pipeline (remyxai/outrider — Claude Code under the hood, with glm-5.2 routed at z.ai's Coding Plan endpoint vs default Opus). The pipeline ran in two modes that probe different parts of the workflow:

• Selection-pass mode (n=9): no pin; each provider freely selects its own paper from the candidate pool. Exercises the full pipeline including selection + verification gates.
• Pin-method mode (n=10): same paper pinned on each fork, both providers run their full chain on identical input. Isolates implementation-side behavior on a forced pick.

The aggregate verdict comes from the n=19 union; the two mode-specific breakdowns below show where the difference comes from.

Reproducing this

The action under test is remyxai/outrider; the remyxai-cli installs the workflow on a target fork and dispatches runs:

# Install Outrider on the target fork (one-time setup).
remyxai outrider init --repo your-fork/repo --interest-id <uuid>

# Drop the alternate provider's API key into the repo's secrets.
remyxai outrider set-provider-secret \
  --repo your-fork/repo --provider zai --key-from ~/zai-key

# Compare the same paper across providers + models.
remyxai outrider trigger --repo your-fork/repo --pin-method 2606.27369v1 \
  --provider anthropic --model claude-opus-4-7
remyxai outrider trigger --repo your-fork/repo --pin-method 2606.27369v1 \
  --provider zai --model glm-5.2

# Or omit --pin-method to let each provider select its own paper.
remyxai outrider trigger --repo your-fork/repo \
  --provider anthropic --model claude-opus-4-7
remyxai outrider trigger --repo your-fork/repo \
  --provider zai --model glm-5.2

--provider picks the company / API endpoint; --model picks the specific model from that provider's catalog.

Headline finding: triage vs attempt

Aggregate outcomes across all 19 paired runs:

Opus triages. When it can ship a PR cleanly, it does (5× more often than GLM). When it can't find a real call site, it exits at selection-pass verification rather than attempting an implementation. The full range of routing outcomes — PR / Issue / skip — gets used roughly in proportion to what the candidate actually warrants: ship, surface for discussion, or drop.

GLM-5.2 tries. GLM rarely exits early — it attempts implementation, and Outrider's downstream gates (preflight, self-review) decide where the work lands. dspy is a clean example: GLM drafted a PR; the self-review pass downgraded it back to an Issue. Opus on the same paper had already exited at selection verification, never starting implementation in the first place. GLM's high Issue-with-context rate (79%) is the pipeline's fallback shape when an attempted implementation doesn't clear validation — not a behavioral preference of GLM's.

The asymmetry isn't "Opus exits more, GLM ships more" — it's Opus triages upstream, GLM lets the pipeline gates resolve it downstream. Both end up on roughly the same outcomes for truly borderline cases; the path through the pipeline is what differs.

The divergence is sharpest at the upstream gates: selection-pass verification (drops all candidates if none are contract-anchored), preflight routing (PR vs Issue before implementation), self-review (downgrades a drafted PR to Issue). The two mode-specific breakdowns below show which gate does which work.

Selection-pass mode: where the divergence appears (n=9)

Without a pinned paper, each provider runs the full pipeline — including the selection-pass verification gate Opus uses to drop candidates that don't have a real call site to wire into.

Two findings stack here:

• Paper-picking converges: both providers picked the same paper from the candidate pool in 6 of 9 runs (67%). Selection is largely model-insensitive — the same candidate pool yields the same picks across providers.
• Same-paper outcomes diverge: within the six same-paper pairs (bolded), 5 of 6 land on different artifact outcomes. Three of those are the skip-verified vs Issue pattern that's specific to this mode — Opus's selection verification drops the candidate; GLM on the same input attempts an Issue.

Aggregate within the mode: 4 of 9 Opus runs exited at skipped_by_selection_verification (vs 2 of 9 GLM); 4 of 9 Opus runs shipped a PR (vs 0 of 9 GLM).

Pin-method mode: where the routing mostly agrees (n=10)

When both providers receive the same paper as forced input, routing matched 8 of 10 paired completions, including the deeper chain gates (high_risk, no_integration, preflight). With selection + verification bypassed, the implementation-side behavior is largely model-insensitive at the routing-decision level.

The one striking implementation-side divergence was atropos: Opus drafted a PR but its self-review downgraded the result to Issue #7; GLM cleared self-review and shipped draft PR #8 — +462 / -1 across 4 files, paper-grounded tests verifying RiVER's two failure modes. The one time GLM shipped a PR in the full bench was when Opus's self-review pulled its own back.

Where their styles consistently differ

The PR-vs-Issue routing direction varies, but how each model writes the artifact given a decision is stable.

GLM: more verbose, more structured, proposes concrete experiments

When GLM produces an Issue, it consistently uses more structure (TL;DR / Suggested experiment / Engineering analysis / What blocks / How to unblock is its house style) and proposes concrete next-step experiments more often than Opus does. In pin-method paired Issues, GLM's body named a specific experiment to run (a profile-then-port for a kernel paper, a BaseMiddleware subclass + conformance test for a control-plane paper, a thin video-eval harness for an active-perception paper). Opus's counterparts on the same papers usually listed scoping questions instead.

Opus: tighter, more file-specific, prefers scoping questions

Opus's Issues consistently name specific files and line numbers, use a tighter prose register, and prefer open scoping questions over proposed experiments. And — the selection-pass signal — Opus is much more willing to skip the run entirely when no candidate clears its verification bar.

Neither style is universally better. GLM's verbose-with-experiments shape gives a maintainer something concrete to react to; Opus's tighter-with-questions shape (or no artifact at all) gives a maintainer either the right scoping prompts to think through or a clean signal to look elsewhere this week.

What each model reaches for

Given the same candidate pool and the same target repo, what does each model do with its exploration budget? Across 17 selection-pass runs, the difference comes through clearest in three places: the shape of the queries each model issues, the shape of the file-reads each model picks, and how each model phrases what it found.

Opus queries by contract; GLM queries by topic

Reading all 44 verbatim refine queries (21 Opus + 21 GLM clean), three patterns dominate:

• GLM defaults to keyword-style topical search. ~60% of GLM's queries are bare theme keywords: "rotated object detection oriented bounding box aerial", "colbert late interaction retrieval reranking unified", "efficient attention token caching diffusion transformer inference". Google-search shaped — high recall on the theme, no opinion about where the result would land in the repo.

• Opus defaults to integration-shape framing. A third of Opus's queries explicitly name how the result would plug in — "VAE latent autoencoder efficient decoding diffusion replacement" (the word replacement signals a swap-in module), "keypoint pose estimation non-maximum suppression skeleton vectorized post-processing" (post-processing names the integration site), "mixture-of-agents self-MoA single-model distillation inference latency aggregation" (binds to a specific repo's existing moa_loop). The query already encodes a hypothesis about where the paper would land.

• Both providers will name specific algorithms when they're confident. On diffusers, Opus issues "LoRA parameter-efficient fine-tuning diffusion transformer FLUX SD3 adapter", GLM issues "parameter efficient fine tuning LoRA diffusion transformer FLUX". Same theme, similar specificity. Where they diverge is in what they do with their other queries — Opus adds the integration-shape token, GLM adds adjacent themes.

The downstream effect: same broad candidate pool, different cuts. When the integration contract is obvious (a scheduler in diffusers/schedulers/, a LoRA modifier in unsloth), both providers find it. When the contract is buried (espnet's CommonPreprocessor, dspy's predict-strategy slot), Opus's anchored framing helps it commit to a verdict or a skip; GLM keeps reading.

Opus reads selectively; GLM reads broadly

GLM opens ~3× more files per run. The reading pattern is the inverse of the query pattern — Opus's contract-shaped queries point it at fewer, more relevant files; GLM's topical queries surface more candidates and the model then reads broadly to find the contract its query didn't specify.

Two boundary cases make the pattern concrete:

• Reading paid off (dspy): GLM's 7 reads located the Module-sibling slot Opus's 2 reads missed. The extra exploration found the integration site Opus's anchored query hadn't reached.
• Reading didn't help (lerobot): GLM read 20 files / 816 lines — the most aggressive file exploration in the bench — and still concluded no candidate fit. The high-band candidates were RL-fine-tuning shapes the repo didn't host; no amount of reading produces a contract that isn't there.

So GLM's reading isn't wasted — sometimes it's load-bearing. But on the median paired run, the additional context didn't change the verdict.

Each model is wrong differently

Two failure modes show up cleanly:

• GLM under uncertainty: confabulates. On opik (observability platform), GLM asserted "the repo layout supplied for routing is empty" — factually wrong; the repo has hundreds of relevant Python files, and Opus on the same input correctly named specific classes. When context-grounding falters, GLM reaches for an unsupported assertion rather than slowing down.

• Opus under uncertainty: under-engages with the meta-question. GLM occasionally catches scope-defining observations that re-frame whether the paper-to-repo mapping even makes sense. On LiveKit Agents (video-understanding paper): "the repo's existing 'action' concept is LLM tool-calling within a voice turn, not perception actions over a video timeline." — a precise dissolution of the paper-to-repo mapping. On AG2 (governance-for-coding-agents paper): "the paper argues governance should NOT be delegated to LLM orchestration, whereas AG2 IS an LLM-orchestration framework — even the intent sits beside, not inside, AG2." — invalidates the recommendation's premise. Opus on the same papers answered the candidate-pool question rather than re-questioning the premise.

Different stress responses to the same prompt. Both can mislead a maintainer; the modes are stylistically distinct enough that you can tell them apart at a glance.

Cost, tokens, speed (combined across both modes, n=17 clean pairs)

Per-run wall-time distribution (n=19 each, all phases):

GLM is slower at every percentile, not just on outliers. Opus's quick low-p exits are the verification skips (sub-3m); its long-tail (~29m) is when the full chain runs on a PR-route. GLM's distribution sits roughly 1.7× longer through the body of the curve.

The 19× combined cost gap is wider than the pin-method-only gap (~11×) because Opus spent more in selection-pass mode — its 4 PR-route runs there each drew on the full chain (~10-30K output tokens per PR), while GLM bailed at Issue and skipped the chain entirely. The token-input gap narrows for the same reason: GLM ran less downstream work because Issue is its dominant exit.

The output-tokens flip — Opus uses more output than GLM in aggregate (1.3×) — is the clearest single number tying back to the headline. Opus shipped 5 PRs total; GLM shipped 1. The pipeline's exit-gate behavior shows up in the output-token totals.

Practical takeaways

1. Selection convergence is the rule. Both providers picked the same paper from the candidate pool 6 of 9 times in selection-pass mode (and routing converged 8 of 10 in pin-method once past selection). For "pick the paper, pick the route" workflows in pin-method mode, either provider is fine.
2. Opus exits at multiple verification gates; GLM tends to continue to artifact production. This is the consistent pattern across both modes. The exit gates are: selection-pass verification (the dominant signal in selection-pass mode), preflight routing (both modes), self-review downgrade on drafted PRs (atropos in pin-method, espnet in selection-pass).
3. The consistent style difference is in shape, not direction. GLM tends to write verbose, structured Issues with concrete proposed experiments; Opus tends to write tighter Issues with file-specific scoping questions — or no artifact at all.
4. Pick the failure mode you'd rather absorb. GLM-5.2 occasionally confabulates under uncertainty (the "empty repo layout" assertion on a 100K-file repo). Opus occasionally under-engages with meta-framing (missing the "this paper sits beside, not inside, your repo" reframings GLM catches). Both can mislead; neither is universally better.

All paired artifacts

For each fork, both the GLM-5.2 and Opus Outrider outputs are public:

Pin-method mode (n=10)

Selection-pass mode (n=9)