Can frontier models master the art of Chinese calligraphy? We challenged them to reproduce 88 of the hardest characters across 20 master calligraphers.
Motivation: Constrained Aesthetic Manipulation
Frontier models have greatly benefited from a wealth of verifiable environments in tool use, math, and coding. These models are extremely good shape rotators, but their capabilities leave much room to be desired in the visual and aesthetic sense. In particular, frontier models are not convincingly capable of constrained aesthetic manipulation — the ability to manipulate and produce aesthetic objects constrained by a human-editable medium (e.g. XML, React, Figma) and references.
To succeed at constrained aesthetic manipulation, agents must master fine-grained tool use, nuanced perception, and long-horizon planning. Here we propose Chinese Calligraphy as a minimally complex aesthetic manipulation task exhibiting all three properties.
Calligraphy-Bench
Motivated by this challenge, we introduce Calligraphy-Bench. Our benchmark consists of 88 characters selected for difficulty (10-25 strokes), each paired with a reference calligraphy image from one of 20 master calligraphers spanning regular, semi-cursive, cursive, and wild cursive styles. Reference images are sourced from the Chinese Calligraphy Styles by Calligraphers dataset on Kaggle (53,000+ images, 200+ calligraphers). Canonical stroke order and geometry come from makemeahanzi, which provides SVG stroke paths and median lines for 9,574 characters derived from the Unihan database.
The model interacts with a canvas through MCP tools. The reference image is provided as a vision message; agents cannot programmatically access its pixels. Otherwise, the agent has access to a workspace and code execution to construct the set of points passed into create_stroke. Agents control stroke position, pressure (thick-thin variation), and size for each stroke. The latest save_document call is submitted as the final calligraphy.
Implementation-wise, we use the OpenHands Python SDK as our agent harness and tldraw as the canvas to render calligraphy.
Calligraphy Rewards
We evaluate along three axes. Stroke Recall measures what fraction of canonical strokes the agent reproduces. Stroke Order checks whether matched strokes were drawn in the correct calligraphic sequence. Visual Fidelity uses a structured rubric where a vision judge lists every visible difference between the reference and the agent's output, citing which of six calligraphic dimensions it belongs to:
Overall style (整体风格) — formal vs casual, elegant vs rough, restrained vs expressive. If one blurs the resolution, are they indistinguishable?
Stroke contrast (笔画粗细) — thick vs. thin strokes, pressure, contrast, stroke smoothness.
Stroke shape (笔画形态) — pointy vs rounded turns, tapered vs blunt endings, curvature, joint sharpness.
Proportions (结构比例) — component sizes, spacing between radicals, width-to-height ratio.
Flow & rhythm (气韵节奏) — how strokes connect and flow, disconnected vs connected, speed impression, baseline steadiness.
Component fidelity (部件准确) — are specific radicals and sub-components correctly formed?
The judge then assigns a verdict from 1–5:
5 — near identical
4 — same style with 1–2 minor diffs
3 — meaningfully different
2 — not the same style but character is correct
1 — character is corrupted
Additionally, we run pairwise head-to-head comparisons between all model outputs using the MJ1 vision judge, producing win rates per character.
Frontier Model Performance
Notably, every model scores below 2.5/5 on visual fidelity, meaning none are able to consistently capture the aesthetic essence of the reference character.
GPT-5.4 leads pairwise win rate at 75%, followed by Opus 4.6 at 60%, Kimi K2.5 at 34%, and GLM-5 at 30%. Opus achieves the highest stroke recall (89%) while GPT-5.4 leads on visual fidelity (2.2/5) and stroke order (82%). In particular, GLM sometimes misidentifies the character it is meant to draw, hallucinating a different character instead of grounding on the reference image.
How Models Approach the Task
Opus 4.6 averages only 2.0 stroke revisions per task, suggesting a plan-then-execute approach. GPT-5.4 produces the most strokes per character (22.7) and is the most willing to clear_canvas and restart. GLM-5 takes the most agent steps (36.5/task) and self-checks most frequently (4.2 captures/task), but produces the fewest strokes (13.5) — spending more time iterating than drawing. Kimi K2.5 has the highest update_stroke rate (7.3/task) with extreme variance across tasks.
GLM-5 uses Chinese calligraphy terminology (横, 竖, 撇, 捺) in its chain of thought but struggles to translate that knowledge into accurate spatial coordinates — and sometimes hallucinates a different character entirely, ignoring the reference image.
Explore the Data
Below, you can see the writing produced by each model across all 88 characters. Each row shows the reference calligraphy alongside every model's attempt, with stroke-by-stroke replay available for each.The full dataset — reference images, model outputs, evaluation scores, and decoded stroke data — is available on HuggingFace.
An Ode to our Ancestors
Last weekend was 清明节, also known as “tomb-sweeping day”. It is a day to mourn, but also honor and celebrate, your ancestors. Calligraphy was one of my grandfather’s favorite past times, and I have very fond memories of watching him in his state of meditative focus. It is probably the earliest example I can recall of what absolute unmixed attention looks like.
Calligraphy-Bench is an ode to him, all of his ancestors, and all of our ancestors. What a privilege it is to be alive right now in this inning of history. I know my grandfather would be amazed and amused and enthralled by all that’s changed and changing.
I hope I make him proud. I hope we make all of our ancestors proud.
Full Links to Calligraphy-Bench
Website: calligraphybench.com
Leaderboard
Dataset
GitHub
Semantic Observability report