Puzzle Solving using Reasoning of Large Language Models: A Survey

TL;DR

This survey examines how large language models perform in puzzle solving, categorizing puzzles into rule-based and rule-less types, and evaluates their reasoning abilities and limitations through various methodologies and benchmarks.

Contribution

It introduces a taxonomy for puzzles and critically assesses LLMs' performance, highlighting challenges and suggesting directions for future research in logical reasoning.

Findings

LLMs show limited performance in complex logical puzzles.

Significant gap between LLM reasoning and human capabilities.

Need for richer datasets and novel strategies for improvement.

Abstract

Exploring the capabilities of Large Language Models (LLMs) in puzzle solving unveils critical insights into their potential and challenges in AI, marking a significant step towards understanding their applicability in complex reasoning tasks. This survey leverages a unique taxonomy -- dividing puzzles into rule-based and rule-less categories -- to critically assess LLMs through various methodologies, including prompting techniques, neuro-symbolic approaches, and fine-tuning. Through a critical review of relevant datasets and benchmarks, we assess LLMs' performance, identifying significant challenges in complex puzzle scenarios. Our findings highlight the disparity between LLM capabilities and human-like reasoning, particularly in those requiring advanced logical inference. The survey underscores the necessity for novel strategies and richer datasets to advance LLMs' puzzle-solving proficiency and contribute to AI's logical reasoning and creative problem-solving advancements.