ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training

TL;DR

ConsistRM is a self-training framework that improves generative reward models by using consistency-aware rewards, reducing reliance on human annotations, and enhancing stability and output consistency.

Contribution

It introduces a novel self-training method with consistency-aware rewards that stabilize training and improve alignment without human-labeled data.

Findings

Outperforms vanilla RFT by 1.5% on benchmark datasets.

Enhances output consistency and reduces position bias.

Provides stable pseudo-labels through temporal consistency.

Abstract

Generative reward models (GRMs) have emerged as a promising approach for aligning Large Language Models (LLMs) with human preferences by offering greater representational capacity and flexibility than traditional scalar reward models. However, GRMs face two major challenges: reliance on costly human-annotated data restricts scalability, and self-training approaches often suffer from instability and vulnerability to reward hacking. To address these issues, we propose ConsistRM, a self-training framework that enables effective and stable GRM training without human annotations. ConsistRM incorporates the Consistency-Aware Answer Reward, which produces reliable pseudo-labels with temporal consistency, thereby providing more stable model optimization. Moreover, the Consistency-Aware Critique Reward is introduced to assess semantic consistency across multiple critiques and allocates fine-grained and differentiated rewards. Experiments on five benchmark datasets across four base models demonstrate that ConsistRM outperforms vanilla Reinforcement Fine-Tuning (RFT) by an average of 1.5%. Further analysis shows that ConsistRM enhances output consistency and mitigates position bias caused by input order, highlighting the effectiveness of consistency-aware rewards in improving GRMs. Our implementation is available at https://github.com/yuliangCarmelo/ConsistRM.