LLM Chess Leaderboard
Simulating chess games between a Random Player and an LLM. Evaluating Chat Models' (1) chess proficiency and (2)
instruction following abilities
__ __ ____ __
/\ \ /\ \ /'\_/`\ /\ _``. /\ \
\ \ \ \ \ \ /\ \ \ \ \/\_\\ \ \___ __ ____ ____
\ \ \ __\ \ \ __\ \ \__\ \ \ \ \/_/_\ \ _ `\ /'__`\ /',__\ /',__\
\ \ \L\ \\ \ \L\ \\ \ \_/\ \ \ \ \L\ \\ \ \ \ \/\ __//\__, `\/\__, `\
\ \____/ \ \____/ \ \_\\ \_\ \ \____/ \ \_\ \_\ \____\/\____/\/\____/
\/___/ \/___/ \/_/ \/_/ \/___/ \/_/\/_/\/____/\/___/ \/___/
__ __ __ __
/\ \ /\ \ /\ "-./ \
\ \ \____ \ \ \____ \ \ \-./\ \
\ \_____\ \ \_____\ \ \_\ \ \_\
\/_____/ \/_____/ \/_/ \/_/
______ __ __ ______ ______ ______
/\ ___\ /\ \_\ \ /\ ___\ /\ ___\ /\ ___\
\ \ \____ \ \ __ \ \ \ __\ \ \___ \ \ \___ \
\ \_____\ \ \_\ \_\ \ \_____\ \/\_____\ \/\_____\
\/_____/ \/_/\/_/ \/_____/ \/_____/ \/_____/
Random Player (White)
♜ ♞ ♝ ♛ ♚ ♝ ♞ ♜ ♟ ♟ ♟ ♟ ♟ ♟ ♟ ♟ · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ♙ ♙ ♙ ♙ ♙ ♙ ♙ ♙ ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖
GAME OVER
- Outcome: Draw
- Max moves reached: 200
- Material White: 16
- Material Black: 18
GPT-4o Mini (Black)
Can Large Language Models play chess? Let's find out ツ
This leaderboard evaluates chess skill and instruction following in an agentic setting: LLMs engage in multi-turn dialogs where they are presented with a choice of actions (e.g., "get board" or "make move") when playing against an opponent (Random Player or Chess Engine).
In 2024, we began with a chaos monkey baseline — a Random Player that chooses legal moves at random. At the time, most models could barely compete and lost either due to an inability to follow game instructions (i.e., hallucinating illegal moves or taking incorrect actions) or by dragging the game to the 200-move limit because they couldn't win.
In 2025, more capable reasoning models nailed both instruction following and chess skill. We've added the Komodo Dragon Chess Engine as a more capable opponent, which is also Elo-rated on chess.com. This allowed us to anchor the results to a real-world rating scale and compute an Elo rating for each model.
Select columns (max 7)
METRICS:
- Player: Model name (playing as Black). Models that also played vs Dragon are marked
with an asterisk in superscript (e.g., 3*).
- Elo: Estimated Elo anchored by Dragon skill levels and calibrated Random. We solve a
1D MLE over aggregated blocks (opponent Elo, wins, draws, losses) and report ±95% CI. When both Random
and Dragon data exist, they are combined. Empty Elo appears for extreme 100% win/loss or no anchored
games.
- Game Duration: Share of maximum game length completed (0-100%); measures
instruction-following stability across many moves. 100% means no games were interrupted due to model
haluscinating moves or actions. 50% means that on average the model boroke the game loop mid-game
(making an average 100 moves out of max 200 allowed)
- Tokens: Completion tokens per move; verbosity/efficiency signal.
- Cost/Elo (main): Estimated cost per 1000 Elo points (Cost/Game divided by Elo, then scaled by 1000). Lower is more cost-efficient.
- Cost/Game (extended): Estimated cost per game based on token usage and model pricing.
ARRANGEMENT & SOURCES:
- Primary sorting: Elo (DESC), then Game Duration (DESC), Tokens (ASC).
- Data sources mix Random-vs-LLM and Dragon-vs-LLM games. Dragon levels map to Elo and provide the
anchor; Random is first calibrated vs Dragon and then used as an opponent for many models.
- Elo ratings are not comparable across player pools, i.e. you can not compare chess.com Elo to FIDE
Elo
- Chess.com references used for context (as of Sep 2025): Rapid Leaderboard (Elo
pool), Magnus Carlsen
stats, and Elo explanation &
player classes.
MATRIX VISUALIZATION:
This plots shows LLM chess players based on two key metrics:
- X-Axis: Game Duration (0-100%) - Shows how well models maintain correct communication protocols throughout the game. Higher values indicate better instruction following ability.
- Y-Axis: Win Rate (0-100%) - The metric is less strict than Win/Loss (Non-Interrupted) used in the leaderboard as it ignores technichal losses due to poor instruction following. Higher values indicate better chess strategy and decision making.
INTERPRETATION:
- Top-Right: Models with both excellent chess skill and instruction following.
- Top-Left: Models with good chess skill but struggle to maintain communication
protocol.
- Bottom-Right: Models that follow instructions well but make poor chess moves.
- Bottom-Left: Models that struggle with both chess strategy and following
instructions.
Libraries and Dependencies Used:
- chess: A Python library for handling chess game rules and basic operations, including
board representation, legal move evaluation, and game state evaluation. This is not a chess engine
running the actual calculation of the best move.
- AG2 (aka Autogen) is used as a backbone for LLM communication. It also implements the
interaction between a Chess Board and custom agents like GameAgent, RandomPlayerAgent, AutoReplyAgent,
and others for simulating different player types.
- Komodo Dragon - the chess engine doing the actual best move calculation, used as a
reference to demonstrate what a real chess player's performance is.
Workflow:
1. The game is initialized with a chess board and two players: a Random Player (as white) and an LLM (as
black).
2. The game loop runs until a termination condition is met, such as checkmate, stalemate, or reaching
the maximum number of moves.
3. Each player takes turns making a move. The Random Player selects a move randomly from the list of
legal moves.
4. The LLM is prompted to make a move using a structured dialog, which includes actions like getting the
current board state, retrieving legal moves, and making a move.
5. The game state is updated after each move, and the board is visualized if enabled.
6. Game statistics are generated and stored at the end of the game.
Dialog:
Here is an example of a dialog that prompts a model to make a move. UCI format refers to Universal Chess Interface notation, which uses source and destination squares (e.g., e2e4 means move from e2 to e4):
Proxy (to Player_Black): You are a professional chess player and you play as black. Now is your turn to make a move. Before making a move you can pick one of the following actions: - 'get_current_board' to get the schema and current status of the board - 'get_legal_moves' to get a UCI formatted list of available moves - 'make_move <UCI formatted move>' when you are ready to complete your turn (e.g., 'make_move e2e4') Respond with the action. Player_Black (to Proxy): get_current_board Proxy (to Player_Black): ♜ ♞ ♝ ♛ ♚ ♝ ♞ ♜ ♟ ♟ ♟ ♟ ♟ ♟ ♟ ♟ · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ♙ ♙ ♙ ♙ ♙ ♙ ♙ ♙ ♖ ♘ ♗ ♕ ♔ ♗ ♘ ♖ Player_Black (to Proxy): get_legal_moves Proxy (to Player_Black): a7a6, b7b6, c7c6, d7d6, e7e6, f7f6, g7g6, h7h6, a7a5, b7b5, c7c5, d7d5, e7e5, f7f5, g7g5, h7h5 Player_Black (to Proxy): make_move e7e5 Proxy (to Player_Black): Move made, switching player
1 move = 1 dialog. The dialog is limited to 10 turns (10 pairs of Proxy/Agent Request/Response messages). A maximum of 3 mistakes (not conforming to communication notation, picking wrong action or wrong move) are allowed per dialog. The game is terminated and LLM is given a loss if the max turns OR number of LLM mistakes is reached in the given dialog. Models that excel at both chess strategy and following instructions precisely will perform best.
GitHub
Paper
Dataset