What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models

TL;DR

This paper identifies the 'Echo Reflection' problem in large language models during complex reasoning tasks and introduces AEPO, a reinforcement learning method, to mitigate this issue and improve reasoning performance.

Contribution

The paper uncovers the causes of Echo Reflection in LLMs and proposes AEPO, a novel reinforcement learning framework with reflection-aware filtration and adaptive entropy optimization.

Findings

AEPO outperforms existing reinforcement learning methods on multiple benchmarks.

Echo Reflection causes LLMs to reiterate without generating new insights.

AEPO effectively reduces echo reflection and enhances reasoning accuracy.

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of reasoning tasks. Recent methods have further improved LLM performance in complex mathematical reasoning. However, when extending these methods beyond the domain of mathematical reasoning to tasks involving complex domain-specific knowledge, we observe a consistent failure of LLMs to generate novel insights during the reflection stage. Instead of conducting genuine cognitive refinement, the model tends to mechanically reiterate earlier reasoning steps without introducing new information or perspectives, a phenomenon referred to as "Echo Reflection". We attribute this behavior to two key defects: (1) Uncontrollable information flow during response generation, which allows premature intermediate thoughts to propagate unchecked and distort final decisions; (2) Insufficient exploration of internal knowledge during reflection, leading to repeating earlier findings rather than generating new cognitive insights. Building on these findings, we proposed a novel reinforcement learning method termed Adaptive Entropy Policy Optimization (AEPO). Specifically, the AEPO framework consists of two major components: (1) Reflection-aware Information Filtration, which quantifies the cognitive information flow and prevents the final answer from being affected by earlier bad cognitive information; (2) Adaptive-Entropy Optimization, which dynamically balances exploration and exploitation across different reasoning stages, promoting both reflective diversity and answer correctness. Extensive experiments demonstrate that AEPO consistently achieves state-of-the-art performance over mainstream reinforcement learning baselines across diverse benchmarks.