Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning

TL;DR

This paper introduces Offline Exploration-Aware fine-tuning (OXA), a novel method that enhances large language models' mathematical reasoning by optimizing data utilization during supervised fine-tuning, leading to improved exploration and performance.

Contribution

The paper proposes OXA, a new fine-tuning approach that promotes better exploration by adjusting data confidence levels, significantly improving reasoning performance over traditional methods.

Findings

OXA achieves an average of +6 Pass@1 and +5 Pass@$k$ improvements.

OXA increases initial policy entropy, fostering better exploration.

Performance gains from OXA persist during extensive RLVR training.

Abstract

Through encouraging self-exploration, reinforcement learning from verifiable rewards (RLVR) has significantly advanced the mathematical reasoning capabilities of large language models. As the starting point for RLVR, the capacity of supervised fine-tuning (SFT) to memorize new chain-of-thought trajectories provides a crucial initialization that shapes the subsequent exploration landscape. However, existing research primarily focuses on facilitating exploration during RLVR training, leaving exploration-aware SFT under-explored. To bridge this gap, we propose Offline eXploration-Aware (OXA) fine-tuning. Specifically, OXA optimizes two objectives: promoting low-confidence verified teacher-distillation data to internalize previously uncaptured reasoning patterns, and suppressing high-confidence incorrect self-distillation data to redistribute probability mass of incorrect patterns toward potentially correct candidates. Experimental results across 6 benchmarks show that OXA consistently improves mathematical reasoning performance, especially achieving an average gain of $+6$ Pass@1 and $+5$ Pass@$k$ points compared to conventional SFT on the Qwen2.5-1.5B-Math. Crucially, OXA elevates initial policy entropy, and performance gains persist throughout extensive RLVR training, demonstrating the long-term value of OXA.