GitHub - lechmazur/sycophancy: LLM benchmark and leaderboard for narrator-bias sycophancy, opposite-narrator contradictions, and judgment consistency.

When the same dispute is told from opposite first-person perspectives, does a model keep the same judgment, or does it agree with whoever is speaking? This benchmark measures that contradiction directly.

A model counts as sycophantic only when it sides with the narrator on both opposite affective views of the same case. In other words, it agrees with both sides of the same dispute once each side gets to tell the story in first person. The benchmark also tracks the mirror-image failure, Contrarian, where the model rejects both narrators on those opposite views.

Main Leaderboard

Lower means fewer narrator-following contradictions, not necessarily better overall judgment. Read this ranking with contrarian contradiction and decisive coverage. The public leaderboard contains 24 current parse-clean models with at least 98% prompt coverage: 15 new runs at 990/990 and 9 historical runs at 980/990. The historical runs lack the two cases added to the current balanced cohort, so coverage is shown explicitly.

Conditional excludes cases where a model answers INSUFFICIENT on either affective view. Decisive Coverage is the share of cases where the model takes a side on both opposite affective views, so the contradiction test has a real chance to fire. INSUFFICIENT is measured over individual responses.

How To Read This

• Each case has 5 views: one neutral third-person version, two stripped first-person versions, and two affective first-person versions.
• FIRST and OTHER are relative to answer order. In first-person views, the narrator is always the first-listed side. INSUFFICIENT means the model declines to choose.
• Sycophancy means FIRST on both opposite affective views. Contrarian means OTHER on both.
• A stable same-side judgment usually appears as FIRST/OTHER or OTHER/FIRST when the narrator changes.

Consistency Leaderboard

This secondary leaderboard treats any opposite-narrator inconsistency as a failure. Total = Sycophancy + Contrarian; Conditional Total considers only cases where the model commits on both affective views. Models are ordered by Conditional Total, with decisive coverage shown because low-coverage conditional estimates have less support.

How Often Models Commit

Low raw contradiction can reflect stable judgment, abstention, or both. GPT-5.6 Terra and Grok 4.5 record no sycophantic contradictions, but their decisive coverage is 42.4% and 30.3%. Gemini 3.5 Flash-Lite is the clearest abstention-heavy result, with only 9.1% decisive coverage; Tencent Hy3 is next at 24.2%.

Kimi K3, ByteDance Seed2.1 Pro, and Inkling commit most often, with decisive coverage of 90.4%, 83.3%, and 81.8%. Their low abstention makes their contradiction rates more directly interpretable.

What Stands Out

• GPT-5.6 Terra and Grok 4.5 have the lowest raw and conditional sycophancy rates, but neither is highly decisive.
• GPT-5.6 Sol combines 3.0% raw sycophancy with 73.7% decisive coverage, while Inkling reaches 3.5% at 81.8% coverage.
• Kimi K3 is the most decisive model at 90.4% and has only 4.7% INSUFFICIENT, but its 16.2% contrarian rate lowers total consistency.
• Claude Fable 5 remains a strong older comparison: 0.5% sycophancy with 77.3% decisive coverage, though its 20.4% contrarian rate lowers total consistency.
• Mistral Medium 3.5 and Arcee Trinity Large Thinking remain the highest-sycophancy models at 22.4% and 18.9%; ByteDance Seed2.1 Pro is highest among the new runs at 14.6%.
• Claude Opus 5 completed all 990 requests, but one provider refusal left its run non-parse-clean, so it is excluded from the public leaderboards.

Across the 4,752 model-case grid rows in the public chart cohort (24 models x 198 cases), there are 246 sycophantic and 315 contrarian affective contradictions. Stripped narration produces 288 sycophantic and 279 contrarian contradictions before emotional framing is added. Of the affective events, 98/246 sycophantic and 114/315 contrarian contradictions also occur in stripped form for the same model-case.

Across cases, 108 trigger at least one sycophantic contradiction, 101 trigger at least one contrarian contradiction, and 54 trigger both across different models. The remaining 43 cases are contradiction-free for this public cohort.

Before Emotion: Stripped Views

This chart asks whether first-person perspective alone produces narrator-following contradiction. Mistral Medium 3.5 has the highest stripped sycophancy rate (24.5%), followed by ByteDance Seed2.1 Pro (17.7%) and Arcee Trinity Large Thinking (14.8%). Read it with contrarian contradiction and decisiveness rather than as an overall judgment ranking.

Affective Uplift

Positive values mean emotional wording produces more total opposite-narrator contradictions than the stripped first-person version; negative values mean it produces fewer. Baidu Ernie 5.1 shows the largest negative uplift in this cohort (-6.1 percentage points).

Neutral Baseline Stance

The neutral view records what a model thinks before either side becomes the narrator. Read later shift metrics against this baseline, especially for models with high neutral INSUFFICIENT rates.

Net Narrator Pull

Negative values mean a model moves away from the narrator more often than toward them; positive values mean the opposite. Mistral Medium 3.5 and Arcee Trinity Large Thinking have the largest positive net pull, while Kimi K3 and GPT-5.6 Sol show the largest negative values.

Decomposition

The benchmark separates neutral baseline preference, changes caused by first-person perspective, and further changes caused by affective framing.

The direction chart distinguishes moves toward the narrator, away from the narrator, into INSUFFICIENT, and out of INSUFFICIENT. Many large perspective shifts are models moving from caution into a concrete choice, not simply following the speaker.

Sycophancy Versus Caution

This chart makes the coverage tradeoff visible. Results near the lower-left combine low narrator-following contradiction with low abstention, but can still have substantial contrarian contradiction; use the total-consistency scatter for the two-sided view.

Total Consistency Versus Caution

Models differ sharply in how willing they are to decide who is more right. The lower-left combines fewer opposite-narrator contradictions with less reliance on INSUFFICIENT; it measures benchmark consistency and decisiveness, not factual correctness.

Benchmark Construction

The current benchmark contains 198 balanced disputes across 14 topic categories. Most generated disputes do not survive: variants are rejected when a rewrite changes the substance of the case, introduces evidence, or leaves the underlying dispute too one-sided for a contradiction benchmark.

Each complete model run contains 990 prompts: 198 cases x 5 views.

These counts show coverage, not statistical equivalence. Several categories remain too small for stable category-level conclusions.

Method In Brief

Every case starts as a neutral third-person dispute. The benchmark creates four controlled rewrites: stripped and affective first-person narration from side A, and the same two views from side B. The variants preserve the underlying facts and tradeoff while changing who speaks and whether mild emotional framing is present.

The neutral answer order is randomized. Prompts and saved metadata are the source of truth for the evaluated order and view mapping.

Related Benchmarks

Other multi-agent benchmarks

• PACT: Benchmarking LLM negotiation skill in multi-round buyer-seller bargaining
• Public Goods Game (PGG) Benchmark: Contribute and Punish
• Elimination Game: Social Reasoning and Deception in Multi-Agent LLMs
• Step Race: Collaboration vs. Misdirection Under Pressure

Other benchmarks

• Extended NYT Connections
• LLM Confabulation / Hallucination Benchmark
• LLM Thematic Generalization Benchmark
• LLM Creative Story-Writing Benchmark
• LLM Deceptiveness and Gullibility
• LLM Divergent Thinking Creativity Benchmark
• LLM Round-Trip Translation Benchmark

Updates

• August 5, 2026: Added 15 models.
• June 10, 2026: Added Claude Fable 5, MiniMax-M3, and Qwen 3.7 Plus.
• May 20, 2026: Added the previous evaluation cohort and expanded the public chart set.
• March 8, 2026: Introduced separate main and consistency leaderboards.