Illusions of reflection: open-ended task reveals systematic failures in Large Language Models' reflective reasoning

TL;DR

This paper critically evaluates whether large language models genuinely perform reflective reasoning by testing their ability to produce and revise scientific test items under real-world constraints, revealing significant limitations in current models.

Contribution

The study introduces an open-ended, rule-constrained task to assess LLMs' reflective reasoning, demonstrating their limited capacity for goal-driven self-correction and the need for external constraint enforcement.

Findings

First-pass performance is often poor, with models producing few valid items.

Reflection yields modest improvements, often due to chance rather than true correction.

Model performance deteriorates with increased open-endedness, showing no advantage for reasoning-optimized models.

Abstract

Humans do not just find mistakes after the fact -- we often catch them mid-stream because 'reflection' is tied to the goal and its constraints. Today's large language models produce reasoning tokens and 'reflective' text, but is it functionally equivalent with human reflective reasoning? Prior work on closed-ended tasks -- with clear, external 'correctness' signals -- can make 'reflection' look effective while masking limits in self-correction. We therefore test eight frontier models on a simple, real-world task that is open-ended yet rule-constrained, with auditable success criteria: to produce valid scientific test items, then revise after considering their own critique. First-pass performance is poor (often zero valid items out of 4 required; mean $\approx$ 1), and reflection yields only modest gains (also $\approx$ 1). Crucially, the second attempt frequently repeats the same violation of constraint, indicating 'corrective gains' arise largely from chance production of a valid item rather than error detection and principled, constraint-sensitive repair. Performance before and after reflection deteriorates as open-endedness increases, and models marketed for 'reasoning' show no advantage. Our results suggest that current LLM 'reflection' lacks functional evidence of the active, goal-driven monitoring that helps humans respect constraints even on a first pass. Until such mechanisms are instantiated in the model itself, reliable performance requires external structure that enforces constraints. Our code is available at: https://github.com/cruiseresearchgroup/LLM_ReflectionTest