LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment

TL;DR

LearnAlign is a gradient-alignment-based data selection method that improves training efficiency for RLVR in LLMs by intelligently choosing representative reasoning data, reducing data needs significantly.

Contribution

It introduces a novel data learnability measure based on success rate to address response-length bias and enhances data efficiency in RLVR training.

Findings

Reduces training data by up to 1,000 data points on GSM8K.

Achieves 77.5% accuracy with less data, outperforming full dataset training.

Demonstrates efficiency on mathematical and code benchmarks.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing LLMs' reasoning abilities, yet its data inefficiency remains a major bottleneck. To address this critical yet challenging issue, we present a novel gradient-alignment-based method, named LearnAlign, which intelligently selects the learnable and representative training reasoning data for RLVR post-training. To overcome the well-known response-length bias in gradient norms, we introduce the data learnability based on the success rate, which indicates the learning potential of each data point. Experiments across five reasoning benchmarks show that our method significantly reduces training data requirements while achieving minor performance degradation or even improving performance compared to full-data training. Specifically, it reduces data requirements by up to 1,000 data points with better performance (77.5%) than that on the full dataset on the GSM8K benchmark (77.0%). Furthermore, its efficiency is demonstrated on both mathematical and code benchmarks by using much less data from the DAPO-MATH-17K dataset.