Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6x6 Sudoku

TL;DR

This study evaluates five Large Language Models on their ability to solve and explain 6x6 Sudoku puzzles, revealing significant challenges in generating strategic and intuitive explanations for collaborative decision-making.

Contribution

It provides an exploratory analysis of LLMs' capabilities in solving and explaining complex puzzles, highlighting gaps in their reasoning and explanation quality.

Findings

One LLM shows limited puzzle-solving success.

None of the LLMs can produce explanations reflecting strategic reasoning.

Results highlight challenges in LLMs' ability to generate trustworthy, tailored explanations.

Abstract

The success of Large Language Models (LLMs) in human-AI collaborative decision-making hinges on their ability to provide trustworthy, gradual, and tailored explanations. Solving complex puzzles, such as Sudoku, offers a canonical example of this collaboration, where clear and customized explanations often hold greater importance than the final solution. In this study, we evaluate the performance of five LLMs in solving and explaining \sixsix{} Sudoku puzzles. While one LLM demonstrates limited success in solving puzzles, none can explain the solution process in a manner that reflects strategic reasoning or intuitive problem-solving. These findings underscore significant challenges that must be addressed before LLMs can become effective partners in human-AI collaborative decision-making.