VerIF: Verification Engineering for Reinforcement Learning in Instruction Following

TL;DR

This paper introduces VerIF, a verification method combining rule-based and LLM-based verification to improve reinforcement learning for instruction-following large language models, achieving state-of-the-art results.

Contribution

The paper proposes VerIF, a novel verification approach for RL in instruction following, supported by a new dataset and demonstrated to enhance model performance.

Findings

Models trained with VerIF outperform previous methods on benchmarks.

VerIF models generalize well to unseen constraints.

The approach maintains overall model capabilities.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a key technique for enhancing large language models (LLMs), with verification engineering playing a central role. However, best practices for RL in instruction following remain underexplored. In this work, we explore the verification challenge in RL for instruction following and propose VerIF, a verification method that combines rule-based code verification with LLM-based verification from a large reasoning model (e.g., QwQ-32B). To support this approach, we construct a high-quality instruction-following dataset, VerInstruct, containing approximately 22,000 instances with associated verification signals. We apply RL training with VerIF to two models, achieving significant improvements across several representative instruction-following benchmarks. The trained models reach state-of-the-art performance among models of comparable size and generalize well to unseen constraints. We further observe that their general capabilities remain unaffected, suggesting that RL with VerIF can be integrated into existing RL recipes to enhance overall model performance. We have released our datasets, codes, and models to facilitate future research at https://github.com/THU-KEG/VerIF.