GitHub - lechmazur/position_bias: A benchmark for testing whether LLM judges keep the same preference when two lightly edited versions of the same story are shown in opposite orders.

LLM Position Bias Benchmark: Swapped-Order Pairwise Judging

When the same two candidate answers are shown in the opposite order, does a model keep the same judgment, or does it move with the display position? This benchmark measures that directly.

This matters because LLMs are increasingly used as graders, evaluators, preference-labelers, and report writers. If the model follows display order, then prompt formatting can quietly become part of the score.

The current results cover 193 verified story pairs, 36 evaluated models, and 386 prompts per full model. The headline result is not a small tie-breaker: across the published results, the model-average first-shown pick rate is 64.3%, and the median model flips its underlying choice in 41.3% of decisive swapped-order case pairs.

Main Leaderboard

This table ranks by Order Flip. Lower means the model more often preserves the same underlying winner across the order swap. Read it together with Decisive Pair Coverage: a low flip rate with many ties is weaker evidence than a low flip rate with high coverage.

How To Read This

• First-Shown Pick is the share of decisive prompt responses where the model chose the version displayed first. A stable model can still land near 50% here because the same preferred story is first in one view and second in the other.
• First Lift is First-Shown Pick - 50%. Positive values mean first-position bias; negative values mean second-position bias.
• Order Flip is the share of decisive two-view case pairs where the model changed its underlying story choice after the order swap. Lower is better.
• Decisive Pair Coverage is the share of cases where the model picked one side in both swapped views. Low coverage means ties are hiding part of the contradiction test.
• Rating Bonus is the average 1-to-7 rating advantage that the same story receives when shown first rather than second. A value near 0 is more order-invariant.

What This Measures

Position bias is a reliability problem for any workflow that asks a model to compare candidates: model evals, writing contests, rubric grading, search/ranking audits, preference-data collection, and product A/B reviews. A model that changes its winner when the candidates are swapped is not only noisy; it can turn arbitrary prompt order into a hidden intervention.

The benchmark isolates that failure mode. It does not ask whether one editor model writes better than another in general. It asks whether an evaluated model can compare two closely related texts while preserving the same underlying preference after the display order changes.

That distinction matters operationally. If position bias is large, a single-pass LLM comparison score should be treated as order-contaminated unless the evaluation randomizes order, uses paired counterbalancing, or aggregates both swapped views. This benchmark makes that contamination visible instead of letting it hide inside a normal-looking win rate.

First-Position Lift

Across the 36-model result set, the model-average first-shown pick rate is 64.3%, with a median of 65.4%. The model-average absolute first-position lift is 15.7 percentage points. So the aggregate pattern is not a subtle tie-breaker: the displayed order materially changes many judgments.

The direction is not universal. Most models prefer the first-shown version too often, but Mistral Large 3 goes the other way, choosing the first-shown version only 27.4% of the time and giving the same story a -0.408 rating shift when shown first.

Rating Effects

The choice and rating metrics tell the same broad story: many models not only choose the first story too often, they also score it higher. The model-average first-position rating bonus is +0.271 on the 1-to-7 rating scale.

The largest rating bonuses are Grok 4.20 0309 (Non-Reasoning) at +0.751, Kimi K2.5 Thinking at +0.648, and GPT-5.4 (high reasoning) at +0.505. That makes Grok non-reasoning a useful warning case: its first-shown pick rate is not the highest in the table, but its numeric ratings move strongly with position.

This scatter separates two failure modes. Some models have a directional first-position pull. Others are unstable under swapping without a huge net first-position lift, because first-position and second-position flips partly cancel.

What Stands Out

• The benchmark finds a large position effect in the current results. The model-average order-flip rate is 43.0%, and the median model flips in 41.3% of decisive two-view cases.
• Mistral Medium 3.5 is the most position-sensitive model in this run: 82.8% first-shown pick rate, +32.8 pp first lift, 72.5% order flip, and +0.492 rating bonus.
• GPT-5.4 (high reasoning) remains highly exposed: 82.3% first-shown pick rate, +32.3 pp first lift, 66.3% order flip, and +0.505 rating bonus.
• The GPT-5.4 family is consistently exposed here. GPT-5.4 no-reasoning is also high at 72.7% first-shown picks and 57.4% order flips, while GPT-5.4 Mini xhigh is at 75.7% and 54.7%.
• Mistral Large 3 is not robust in the opposite direction. It has the strongest second-position bias: 27.4% first-shown picks, -22.6 pp lift, and 58.7% order flips.
• Claude Opus 4.8 (high reasoning) and Gemini 3.5 Flash are among the best new additions by this metric: Opus 4.8 has 27.4% order flips with 92.7% decisive-pair coverage; Gemini 3.5 Flash has 29.8% order flips with 92.2% coverage.
• ByteDance Seed2.0 Pro and DeepSeek V3.2 remain clean low-bias results among models with decent coverage: Seed has 28.0% order flips, 83.4% decisive-pair coverage, and only -1.9 pp first lift; DeepSeek has 30.3% order flips, 75.1% coverage, and +3.7 pp lift.
• Xiaomi MiMo V2 Pro has the lowest order-flip rate at 19.8%, but it also has only 54.9% decisive-pair coverage and a 30.1% tie-prompt rate. That looks more like cautious stability than clean order-invariant judgment.
• Low net first-position lift is not enough by itself. Qwen3.5-122B-A10B is only +6.7 pp on first lift, but still flips its underlying story choice in 51.7% of decisive case pairs.
• Ties are the main non-decisive behavior. The model-average tie-prompt rate is 14.4%; INSUFFICIENT is 0.0%; refusals total only 4 prompt responses across the results.
• The story-pair design is intentionally narrow in the current results: all 193 verified story pairs compare sibling edits from Claude Sonnet 4.6 (high reasoning) and GPT-5.4 (high reasoning), with slot assignment randomized by case. These results are about sensitivity to order over that controlled set of edits, not a universal statement about every possible answer pair.

Most Order-Sensitive Cases

Some story pairs are much more order-sensitive than others. The strongest case-level signal is not just "models disagree"; it is that the same models change which underlying story version they prefer when the pair is displayed in the opposite order.

storm drain tennis ball is a useful caveat: it reaches 100.0% order flip only because very few model-case pairs are decisive there. observatory red flashlight, lighthouse oil room, parking garage puddle, rooftop violin lesson, and observatory visitor log are more informative high-sensitivity examples because many more models commit on both views.

Worked Example

A-first prompt:
<rating_first>6</rating_first>
<rating_other>5</rating_other>
<answer>1</answer>

B-first prompt:
<rating_first>6</rating_first>
<rating_other>5</rating_other>
<answer>1</answer>

A-first prompt:
<rating_first>6</rating_first>
<rating_other>5</rating_other>
<answer>1</answer>

B-first prompt:
<rating_first>6</rating_first>
<rating_other>7</rating_other>
<answer>2</answer>

Outcome Mix

This chart decomposes each model's two-view case outcomes into stable choices, position-following flips, and unresolved cases. It is the fastest way to see why the same headline flip rate can mean different things: one model may flip because it follows the displayed first story, while another may have many ties or a second-position pull.

The coverage view is especially important for this benchmark. A model that refuses to choose cannot contradict itself, but that is not the same as being order-invariant.

Benchmark Construction

1. Generate a short base story plus a bounded change request.
2. Ask two editor models to apply the same change request, with Story A and Story B assigned per case rather than tied to one fixed editor.
3. Verify that each story version is still the same core story and that the requested change was applied.
4. Ask each evaluated model which version is better in both display orders.
5. Parse tag-style responses and compute first-position lift, order-flip rate, and first-position rating bonus.

The current results use:

Writer model distribution:

Topic category coverage is mostly general-purpose story situations: 188 general cases, 3 planning cases, 1 reasoning case, and 1 high-stakes case. Those small non-general buckets are included for coverage transparency only and are not stable category-level findings.

Method In Brief

The benchmark is deliberately narrower than a general "which model writes better?" test. The goal is to isolate order sensitivity in pairwise judging.

The evaluated model sees:

• the original assignment
• two sibling story versions
• answer labels such as 1 and 2
• rating tags for the first-shown and second-shown story

The evaluated model does not see:

• the hidden edit request
• which editor produced which story version
• the fact that the same pair will later appear in the opposite order

Responses are read from tags rather than strict JSON. The required response tags are:

<rating_first>1..7</rating_first>
<rating_other>1..7</rating_other>
<answer>1|2|TIE|INSUFFICIENT</answer>

For scoring, answers are mapped back to the underlying story. Choosing 1 in the A-first prompt and choosing 2 in the B-first prompt both count as the same stable preference for Story A.

Data And Reports

• Summary report
• Data guide
• Data file list
• Prompt index
• Model answers
• Model answers with response text
• Model metrics
• Case metrics

Related Benchmarks

• LLM Sycophancy Benchmark - opposite-narrator contradictions and judgment consistency.
• LLM Debate Benchmark - sustained adversarial argument under active opposition.
• LLM Persuasion Benchmark - multi-turn position movement and target susceptibility.
• Buyout Game Benchmark - multi-agent bargaining, transfers, and endgame wealth management.
• PACT - multi-round buyer-seller bargaining with hidden private values.
• BAZAAR - economic decision-making in a competitive simulated market.
• Elimination Game - social reasoning and deception in a multi-agent elimination setting.
• Step Race - collaboration versus misdirection under time pressure.
• LLM Round-Trip Translation Benchmark - meaning and voice preservation through round-trip translation.
• LLM Creative Story-Writing Benchmark - creative brief following and story quality.
• Extended NYT Connections - larger Connections-style puzzles with extra trick words.
• LLM Confabulation / Hallucination Benchmark - factual reliability and invented-answer behavior.
• LLM Thematic Generalization Benchmark - inferring a latent theme from examples and anti-examples.
• LLM Deceptiveness and Gullibility - deception, gullibility, and adversarial claim handling.

Updates

• June 11, 2026: Added nine newer evaluated models and refreshed the published results.
• April 21, 2026: Initial version.