LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess

TL;DR

LLM CHESS is a new benchmarking framework that evaluates large language models' reasoning and instruction-following abilities through extended chess interactions, revealing significant performance gaps and robustness challenges.

Contribution

This work introduces LLM CHESS, a novel dynamic benchmark for assessing reasoning and instruction-following in LLMs via chess, including a ranking system and public resources.

Findings

Many state-of-the-art models struggle to complete games or win consistently.

A clear separation exists between reasoning and non-reasoning models.

The benchmark's stochastic nature reduces overfitting and prevents saturation.

Abstract

We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extended agentic interaction in the domain of chess. We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including win and loss rates, move quality, move legality, hallucinated actions, and game duration. For a subset of top reasoning models, we derive an Elo estimate by playing against a chess engine with variably configured skill, which allows for comparisons between models in an easily understandable way. Despite the simplicity of the instruction-following task and the weakness of the opponent, many state-of-the-art models struggle to complete games or achieve consistent wins. Similar to other benchmarks on complex reasoning tasks, our experiments reveal a clear separation between reasoning and non-reasoning models. However, unlike existing static benchmarks, the stochastic and dynamic nature of LLM CHESS uniquely reduces overfitting and memorization while preventing benchmark saturation, proving difficult even for top reasoning models. To support future work on evaluating reasoning and instruction-following in LLMs, we release our experimental framework, a public leaderboard, and a dataset of associated games.