Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring

TL;DR

This paper assesses large language models' systematic reasoning abilities using graph coloring problems, revealing progress and limitations in their problem-solving accuracy and reliability across different problem complexities.

Contribution

It introduces a novel benchmarking approach using graph coloring to evaluate LLM reasoning, highlighting the models' strengths and weaknesses in structured problem-solving.

Findings

Models exhibit >60% error on difficult problems in all frames.

No model achieves perfect accuracy on simple 2-coloring problems.

Framing effects significantly influence model performance.

Abstract

Contemporary large language models are powerful problem-solving tools, but they exhibit weaknesses in their reasoning abilities which ongoing research seeks to mitigate. We investigate graph coloring as a means of evaluating an LLM's capacities for systematic step-by-step reasoning and possibility space exploration, as well as effects of semantic problem framing. We test Claude 3.5 Sonnet, Llama 3.1 405B, Gemini 1.5 Pro, GPT-4o, o1-mini, and DeepSeek-R1 on a dataset of $k$-coloring problems with $2 \leq k \leq 4$ and vertex count $4 \leq n \leq 8$, using partial algorithmic solvers to further categorize problems by difficulty. In addition to substantial but varying framing effects, we find that all models except o1-mini and R1 exhibit $>60\%$ error rates on difficult problem types in all frames ($>15\%$ for o1-mini and $>10\%$ for R1), and no model achieves perfect accuracy even in the simple domain of 2-coloring 4-vertex graphs. Our results highlight both the considerable recent progress in LLM systematic reasoning and the limits of its reliability, especially in relation to increasing computational costs. We expect that more complex graph coloring problems, and procedural generation of arbitrary-complexity reasoning problems more broadly, offer further untapped potential for LLM benchmarking.