Precision over Diversity: High-Precision Reward Generalizes to Robust Instruction Following

TL;DR

This paper demonstrates that high-precision rewards, focusing on accuracy rather than diversity, lead to better instruction-following performance and robustness in large language models, challenging the traditional emphasis on diverse training data.

Contribution

The study shows that training with hard-only, high-precision rewards outperforms mixed datasets, proposing a data refinement strategy that enhances efficiency and generalization in instruction following.

Findings

High-precision rewards improve model performance by 13.4%.

Training with hard-only constraints reduces training time by 58%.

Reward precision is more critical than constraint diversity for effective alignment.

Abstract

A central belief in scaling reinforcement learning with verifiable rewards for instruction following (IF) tasks is that, a diverse mixture of verifiable hard and unverifiable soft constraints is essential for generalizing to unseen instructions. In this work, we challenge this prevailing consensus through a systematic empirical investigation. Counter-intuitively, we find that models trained on hard-only constraints consistently outperform those trained on mixed datasets. Extensive experiments reveal that reward precision, rather than constraint diversity, is the primary driver of effective alignment. The LLM judge suffers from a low recall rate in detecting false response, which leads to severe reward hacking, thereby undermining the benefits of diversity. Furthermore, analysis of the attention mechanism reveals that high-precision rewards develop a transferable meta-skill for IF. Motivated by these insights, we propose a simple yet effective data-centric refinement strategy that prioritizes reward precision. Evaluated on five benchmarks, our approach outperforms competitive baselines by 13.4\% in performance while achieving a 58\% reduction in training time, maintaining strong generalization beyond instruction following. Our findings advocate for a paradigm shift: moving away from the indiscriminate pursuit of data diversity toward high-precision rewards.