Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skills

TL;DR

This study evaluates ChatGPT's ability to understand and reason about chess, revealing its strengths in strategy coherence and decision-making, as well as limitations in formal language comprehension and self-regulation.

Contribution

It provides a comprehensive assessment of ChatGPT's performance on formal reasoning tasks like chess, highlighting specific limitations and potential areas for improvement.

Findings

ChatGPT demonstrates strategic coherence in chess gameplay.

Model's decision-making improves with more natural language context.

Limitations in attention mechanisms affect formal language understanding.

Abstract

While large language models have made strides in natural language processing, their proficiency in complex reasoning tasks requiring formal language comprehension, such as chess, remains less investigated. This paper probes the performance of ChatGPT, a sophisticated language model by OpenAI in tackling such complex reasoning tasks, using chess as a case study. Through robust metrics examining both the legality and quality of moves, we assess ChatGPT's understanding of the chessboard, adherence to chess rules, and strategic decision-making abilities. Our evaluation identifies limitations within ChatGPT's attention mechanism that affect its formal language comprehension and uncovers the model's underdeveloped self-regulation abilities. Our study also reveals ChatGPT's propensity for a coherent strategy in its gameplay and a noticeable uptick in decision-making assertiveness when the model is presented with a greater volume of natural language or possesses a more lucid understanding of the state of the chessboard. These findings contribute to the growing exploration of language models' abilities beyond natural language processing, providing valuable information for future research towards models demonstrating human-like cognitive abilities.