Beyond Memorization: Distinguishing between Reductive and Epistemic Reasoning in LLMs using Classic Logic Puzzles

TL;DR

This paper investigates how large language models perform epistemic reasoning on classic logic puzzles, distinguishing between memorization, reduction, and genuine reasoning, revealing limitations in current models.

Contribution

It introduces a reduction ladder to analyze how models handle epistemic puzzles, highlighting their struggles with true epistemic reasoning beyond memorization and reduction.

Findings

Large models succeed via reduction in some cases

Models fail early on when reduction is difficult

All models struggle with genuine epistemic reasoning

Abstract

Epistemic reasoning requires agents to infer the state of the world from partial observations and information about other agents' knowledge. Prior work evaluating LLMs on canonical epistemic puzzles interpreted their behavior through a dichotomy between epistemic reasoning and brittle memorization. We argue that this framing is incomplete: in recent models, memorization is better understood as a special case of reduction, where a new instance is mapped onto a known problem. Instead, we introduce a reduction ladder, a sequence of modifications that progressively move instances away from a canonical epistemic puzzle, making reduction increasingly difficult while preserving the underlying logic. We find that while some large models succeed via reduction, other models fail early, and all models struggle once epistemic reasoning is required.