Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection

TL;DR

The paper introduces ORION, a framework that refines reasoning traces for small language models by incorporating their errors, leading to improved mathematical reasoning performance and more coherent reasoning chains.

Contribution

It proposes an error-aware self-reflection method to enhance reasoning distillation from long-form chains-of-thought for small language models.

Findings

ORION improves performance by over 2% on reasoning benchmarks.

Refined CoTs exhibit higher coherence and logical consistency.

The method effectively utilizes student errors to tailor reasoning traces.

Abstract

Large Language Models (LLMs) have exhibited strong reasoning capabilities and achieved remarkable performance in mathematical problem-solving tasks. Recently, distilling reasoning ability from long-form Chains-of-Thought (CoTs) has emerged as a promising approach for enhancing Small Language Models (SLMs). Existing studies typically treat SLMs as student models and use long-form CoTs as supervision signals for Supervised Fine-Tuning (SFT) to transfer reasoning ability. However, such long-form CoT teachers are usually unaware of the student model's capacity, which limits the effective utilization of the provided reasoning traces. To overcome this limitation, we propose errOr-aware self-ReflectION (ORION), a framework that refines teacher CoTs through an Error-Aware Reflection process. ORION enables the student model to construct more tailored teacher CoTs by refining teacher CoTs and incorporating its own reasoning errors. Experiments on multiple mathematical reasoning benchmarks demonstrate that ORION consistently improves performance by more than 2% over all baselines. Further analysis reveals that the CoTs constructed by ORION exhibit higher coherence and logical consistency, thereby serving as more effective supervision signals for SFT. All codes are available at https://github.com/NEUIR/ORION.git.