Explore the Reasoning Capability of LLMs in the Chess Testbed

TL;DR

This paper enhances large language models' reasoning in chess by integrating expert-annotated strategy and tactics, demonstrating improved move selection and the benefit of language explanations.

Contribution

It introduces the MATE dataset with expert annotations and finetunes LLaMA-3-8B to improve chess reasoning, outperforming existing models.

Findings

Finetuned LLaMA-3-8B surpasses GPT, Claude, and Gemini in chess move selection.

Language explanations improve reasoning capabilities of LLMs.

Expert-annotated strategy and tactic data enhances model performance.

Abstract

Reasoning is a central capability of human intelligence. In recent years, with the advent of large-scale datasets, pretrained large language models have emerged with new capabilities, including reasoning. However, these models still struggle with long-term, complex reasoning tasks, such as playing chess. Based on the observation that expert chess players employ a dual approach combining long-term strategic play with short-term tactical play along with language explanation, we propose improving the reasoning capability of large language models in chess by integrating annotated strategy and tactic. Specifically, we collect a dataset named MATE, which consists of 1 million chess positions with candidate moves annotated by chess experts for strategy and tactics. We finetune the LLaMA-3-8B model and compare it against state-of-the-art commercial language models in the task of selecting better chess moves. Our experiments show that our models perform better than GPT, Claude, and Gemini models. We find that language explanations can enhance the reasoning capability of large language models.