GitHub - lechmazur/position_bias: A benchmark for testing whether LLM judges keep the same preference when two lightly edited versions of the same story are shown in opposite orders.

LLM Position Bias Benchmark: Swapped-Order Pairwise Judging

When the same two candidate answers are shown in the opposite order, does a model keep the same judgment, or does it move with the display position? This benchmark measures that directly.

This matters because LLMs are increasingly used as graders, evaluators, preference-labelers, and report writers. If a judge model follows display order, then prompt formatting can quietly become part of the score.

The current public snapshot is: 193 verified story pairs, 27 judge models, and 386 prompts per full model. The headline result is not a small tie-breaker: across the report view, the model-average first-shown pick rate is 63.3%, and the median model flips its underlying choice in 44.8% of decisive swapped-order case pairs.

Main Leaderboard

This table ranks by Order Flip. Lower means the model more often preserves the same underlying winner across the order swap. Read it together with Decisive Pair Coverage: a low flip rate with many ties is weaker evidence than a low flip rate with high coverage.

How To Read This

• First-Shown Pick is the share of decisive prompt responses where the model chose the version displayed first. A stable judge can still land near 50% here because the same preferred story is first in one view and second in the other.
• First Lift is First-Shown Pick - 50%. Positive values mean first-position bias; negative values mean second-position bias.
• Order Flip is the share of decisive two-view case pairs where the model changed its underlying canonical choice after the order swap. Lower is better.
• Decisive Pair Coverage is the share of cases where the model picked one side in both swapped views. Low coverage means ties are hiding part of the contradiction test.
• Rating Bonus is the average 1-to-7 rating advantage that the same story receives when shown first rather than second. A value near 0 is more order-invariant.

What This Measures

Position bias is a reliability problem for any workflow that asks a model to compare candidates: model evals, writing contests, rubric grading, search/ranking audits, preference-data collection, and product A/B reviews. A judge that changes its winner when the candidates are swapped is not only noisy; it can turn arbitrary prompt order into a hidden intervention.

The benchmark isolates that failure mode. It does not ask whether one editor model writes better than another in general. It asks whether a judge can compare two closely related texts while preserving the same underlying preference after the display order changes.

That distinction matters operationally. If position bias is large, a single-pass LLM judge score should be treated as order-contaminated unless the evaluation randomizes order, uses paired counterbalancing, or aggregates both swapped views. This benchmark makes that contamination visible instead of letting it hide inside a normal-looking win rate.

First-Position Lift

Across the 27-model report view, the model-average first-shown pick rate is 63.3%, with a median of 65.6%. The model-average absolute first-position lift is 15.2 percentage points. So the aggregate pattern is not a subtle tie-breaker: the displayed order materially changes many judgments.

The direction is not universal. Most models prefer the first-shown version too often, but Mistral Large 3 goes the other way, choosing the first-shown version only 27.4% of the time and giving the same story a -0.408 rating shift when shown first.

Rating Effects

The choice and rating metrics tell the same broad story: many models not only choose the first story too often, they also score it higher. The model-average first-position rating bonus is +0.257 on the 1-to-7 rating scale.

The largest rating bonuses are Grok 4.20 0309 (Non-Reasoning) at +0.751, Kimi K2.5 Thinking at +0.648, and GPT-5.4 (high reasoning) at +0.505. That makes Grok non-reasoning a useful warning case: its first-shown pick rate is not the highest in the table, but its numeric ratings move strongly with position.

This scatter separates two failure modes. Some models have a directional first-position pull. Others are unstable under swapping without a huge net first-position lift, because first-position and second-position flips partly cancel.

What Stands Out

• The benchmark finds a large position effect in the current snapshot. The model-average order-flip rate is 44.0%, and the median model flips in 44.8% of decisive two-view cases.
• GPT-5.4 (high reasoning) is the most position-sensitive model in this run: 82.3% first-shown pick rate, +32.3 pp first lift, 66.3% order flip, and +0.505 rating bonus.
• The GPT-5.4 family is consistently exposed here. GPT-5.4 no-reasoning is also high at 72.7% first-shown picks and 57.4% order flips, while GPT-5.4 Mini xhigh is at 75.7% and 54.7%.
• Mistral Large 3 is not robust in the opposite direction. It has the strongest second-position bias: 27.4% first-shown picks, -22.6 pp lift, and 58.7% order flips.
• ByteDance Seed2.0 Pro and DeepSeek V3.2 are the cleanest low-bias results among models with decent coverage: Seed has 28.0% order flips, 83.4% decisive-pair coverage, and only -1.9 pp first lift; DeepSeek has 30.3% order flips, 75.1% coverage, and +3.7 pp lift.
• Xiaomi MiMo V2 Pro has the lowest order-flip rate at 19.8%, but it also has only 54.9% decisive-pair coverage and a 30.1% tie-prompt rate. That looks more like cautious stability than clean order-invariant judgment.
• Low net first-position lift is not enough by itself. Qwen3.5-122B-A10B is only +6.7 pp on first lift, but still flips canonical choice in 51.7% of decisive case pairs.
• Ties are the main non-decisive behavior. The model-average tie-prompt rate is 14.7%; INSUFFICIENT is 0.0%; refusals total only 4 prompt responses across the snapshot.
• The source-pair design is intentionally narrow in this snapshot: all 193 accepted story pairs compare sibling edits from Claude Sonnet 4.6 (high reasoning) and GPT-5.4 (high reasoning), with slot assignment randomized by case. These results are about judge sensitivity to order over that controlled sibling-edit surface, not a universal statement about every possible answer pair.

Most Order-Sensitive Cases

Some story pairs are much more order-sensitive than others. The strongest case-level signal is not just "models disagree"; it is that the same models change which underlying variant they prefer when the pair is displayed in the opposite order.

storm drain tennis ball is a useful caveat: it reaches 100.0% order flip only because very few model-case pairs are decisive there. underpass mural cleanup, lighthouse oil room, midnight bakery, observatory red flashlight, and parking garage puddle are more informative high-sensitivity examples because many more judges commit on both views.

Worked Example

variant_a_first:
<rating_first>6</rating_first>
<rating_other>5</rating_other>
<answer>1</answer>

variant_b_first:
<rating_first>6</rating_first>
<rating_other>5</rating_other>
<answer>1</answer>

variant_a_first:
<rating_first>6</rating_first>
<rating_other>5</rating_other>
<answer>1</answer>

variant_b_first:
<rating_first>6</rating_first>
<rating_other>7</rating_other>
<answer>2</answer>

Outcome Mix

This chart decomposes each model's two-view case outcomes into stable choices, position-following flips, and unresolved cases. It is the fastest way to see why the same headline flip rate can mean different things: one model may flip because it follows the displayed first story, while another may have many ties or a second-position pull.

The coverage view is especially important for this benchmark. A model that refuses to choose cannot contradict itself, but that is not the same as being order-invariant.

Benchmark Construction

1. Generate a short base story plus a bounded change request.
2. Ask two editor models to apply the same change request, with variant_a and variant_b assigned per case rather than tied to one fixed editor.
3. Verify that each variant is still the same core story and that the requested change was applied.
4. Ask judge models which variant is better in both display orders.
5. Parse tag-based judge responses and compute first-position lift, order-flip rate, and first-position rating bonus.

The current snapshot uses:

Writer model distribution:

Topic category coverage is mostly general-purpose story situations: 188 general cases, 3 planning cases, 1 reasoning case, and 1 high-stakes case. Those small non-general buckets are included for coverage transparency only and are not stable category-level findings.

Method In Brief

The benchmark is deliberately narrower than a general "which model writes better?" test. The goal is to isolate order sensitivity in pairwise judging.

The judge sees:

• the original assignment
• two sibling story variants
• answer labels such as 1 and 2
• rating tags for the first-shown and second-shown story

The judge does not see:

• the hidden edit request
• which editor produced which variant
• the fact that the same pair will later appear in the opposite order

The parser expects tag-style outputs rather than strict JSON. The required pass-1 tags are:

<rating_first>1..7</rating_first>
<rating_other>1..7</rating_other>
<answer>1|2|TIE|INSUFFICIENT</answer>

Internally, answers are normalized to canonical variant IDs, so choosing 1 in the variant_a_first prompt and choosing 2 in the variant_b_first prompt both count as the same stable underlying preference for variant_a.

Artifacts

• Generated summary
• Interactive chart index
• Public data bundle guide
• Public data manifest
• Prompt index
• Parsed model answers
• Parsed model answers with raw response text
• Model metrics
• Case metrics

Related Benchmarks

• LLM Sycophancy Benchmark - opposite-narrator contradictions and judgment consistency.
• LLM Debate Benchmark - sustained adversarial argument under active opposition.
• LLM Persuasion Benchmark - multi-turn position movement and target susceptibility.
• Buyout Game Benchmark - multi-agent bargaining, transfers, and endgame wealth management.
• PACT - multi-round buyer-seller bargaining with hidden private values.
• BAZAAR - economic decision-making in a competitive simulated market.
• Elimination Game - social reasoning and deception in a multi-agent elimination setting.
• Step Race - collaboration versus misdirection under time pressure.
• LLM Round-Trip Translation Benchmark - meaning and voice preservation through round-trip translation.
• LLM Creative Story-Writing Benchmark - creative brief following and story quality.
• Extended NYT Connections - larger Connections-style puzzles with extra trick words.
• LLM Confabulation / Hallucination Benchmark - factual reliability and invented-answer behavior.
• LLM Thematic Generalization Benchmark - inferring a latent theme from examples and anti-examples.
• LLM Deceptiveness and Gullibility - deception, gullibility, and adversarial claim handling.

Updates

• April 21, 2026: Initial version.