Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles

TL;DR

This paper reviews how visual puzzles serve as diagnostic tools for assessing reasoning in large vision-language models, highlighting current limitations and proposing future directions for more reasoning-aware systems.

Contribution

It provides a unified framework for understanding visual puzzle reasoning in LVLMs and links puzzle design to cognitive reasoning mechanisms, identifying key limitations in current models.

Findings

Current models show brittle generalization.

Perception and reasoning are tightly entangled.

There is a gap between explanations and faithful execution.

Abstract

Puzzles have long served as compact and revealing probes of human cognition, isolating abstraction, rule discovery, and systematic reasoning with minimal reliance on prior knowledge. Leveraging these properties, visual puzzles have recently emerged as a powerful diagnostic tool for evaluating the reasoning abilities of Large Vision-Language Models (LVLMs), offering controlled, verifiable alternatives to open-ended multimodal benchmarks. This survey provides a unified perspective of visual puzzle reasoning in LVLMs. We frame visual puzzles through a common abstraction and organize existing benchmarks by the reasoning mechanisms they target (inductive, analogical, algorithmic, deductive, and geometric/spatial), thereby linking puzzle design to the cognitive operations required for solving. Synthesizing empirical evidence across these categories, we identify consistent limitations in current models, including brittle generalization, tight entanglement between perception and reasoning, and a persistent gap between fluent explanations and faithful execution. By framing visual puzzles as diagnostic instruments rather than task formats, this survey elaborates on the state of LVLM reasoning and outlines key directions for future benchmarks and reasoning-aware multimodal systems.