Teaching Large Reasoning Models Effective Reflection

TL;DR

This paper introduces SCFT and RLERR, novel training methods that improve large reasoning models by fostering effective self-reflection, leading to better reasoning accuracy and reflection quality on challenging benchmarks.

Contribution

The paper presents SCFT and RLERR, innovative techniques that enhance the reflective reasoning capabilities of LRMs through self-generated critiques and reinforcement learning.

Findings

SCFT improves critique quality and reasoning accuracy.

RLERR guides models to internalize self-correction effectively.

Methods outperform state-of-the-art baselines on AIME benchmarks.

Abstract

Large Reasoning Models (LRMs) have recently shown impressive performance on complex reasoning tasks, often by engaging in self-reflective behaviors such as self-critique and backtracking. However, not all reflections are beneficial-many are superficial, offering little to no improvement over the original answer and incurring computation overhead. In this paper, we identify and address the problem of superficial reflection in LRMs. We first propose Self-Critique Fine-Tuning (SCFT), a training framework that enhances the model's reflective reasoning ability using only self-generated critiques. SCFT prompts models to critique their own outputs, filters high-quality critiques through rejection sampling, and fine-tunes the model using a critique-based objective. Building on this strong foundation, we further introduce Reinforcement Learning with Effective Reflection Rewards (RLERR). RLERR leverages the high-quality reflections initialized by SCFT to construct reward signals, guiding the model to internalize the self-correction process via reinforcement learning. Experiments on two challenging benchmarks, AIME2024 and AIME2025, show that SCFT and RLERR significantly improve both reasoning accuracy and reflection quality, outperforming state-of-the-art baselines. All data and codes are available at https://github.com/wanghanbinpanda/SCFT.