ChessQA: Evaluating Large Language Models for Chess Understanding

TL;DR

ChessQA is a comprehensive benchmark designed to evaluate large language models' understanding of chess across multiple skill levels and task categories, providing a detailed and evolving assessment framework.

Contribution

The paper introduces ChessQA, a new benchmark that systematically measures LLMs' chess understanding across five categories, surpassing previous narrow evaluations.

Findings

Persistent weaknesses across all categories in current LLMs

Evaluation reveals varied performance by model size and training methods

Provides a platform for ongoing research and model improvement

Abstract

Chess provides an ideal testbed for evaluating the reasoning, modeling, and abstraction capabilities of large language models (LLMs), as it has well-defined structure and objective ground truth while admitting a wide spectrum of skill levels. However, existing evaluations of LLM ability in chess are ad hoc and narrow in scope, making it difficult to accurately measure LLM chess understanding and how it varies with scale, post-training methodologies, or architecture choices. We present ChessQA, a comprehensive benchmark that assesses LLM chess understanding across five task categories (Structural, Motifs, Short Tactics, Position Judgment, and Semantic), which approximately correspond to the ascending abstractions that players master as they accumulate chess knowledge, from understanding basic rules and learning tactical motifs to correctly calculating tactics, evaluating positions, and semantically describing high-level concepts. In this way, ChessQA captures a more comprehensive picture of chess ability and understanding, going significantly beyond the simple move quality evaluations done previously, and offers a controlled, consistent setting for diagnosis and comparison. Furthermore, ChessQA is inherently dynamic, with prompts, answer keys, and construction scripts that can evolve as models improve. Evaluating a range of contemporary LLMs, we find persistent weaknesses across all five categories and provide results and error analyses by category. We will release the code, periodically refreshed datasets, and a public leaderboard to support further research.