Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning

TL;DR

PieceHint is a framework that improves reinforcement learning for language models by strategically providing critical reasoning hints, balancing guidance and independence to enhance reasoning diversity and performance.

Contribution

It introduces a method to score and selectively provide hints during training, enabling models to transition from guided learning to independent reasoning.

Findings

Achieves comparable performance to larger models on mathematical benchmarks.

Preserves pass@k diversity across different k values.

Effectively balances hint guidance and reasoning diversity.

Abstract

Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma: training on easy problems can cause overfitting and pass@k degradation, while training on hard problems often results in sparse rewards. Recent question augmentation methods address this by prepending partial solutions as hints. However, uniform hint provision may introduce redundant information while missing critical reasoning bottlenecks, and excessive hints can reduce reasoning diversity, causing pass@k degradation. We propose \textbf{PieceHint}, a hint injection framework that strategically identifies and provides critical reasoning steps during training. By scoring the importance of different reasoning steps, selectively allocating hints based on problem difficulty, and progressively withdrawing scaffolding, PieceHint enables models to transition from guided learning to independent reasoning. Experiments on six mathematical reasoning benchmarks show that our 1.5B model achieves comparable average performance to 32B baselines while preserving pass@k diversity across all $k$ values.