CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling

TL;DR

This paper introduces CDRRM, a novel framework for generating interpretable and high-quality rubrics for reward modeling in LLMs, addressing bias and scalability issues with a contrast-then-synthesis approach.

Contribution

The paper presents a contrast-then-synthesis paradigm for rubric generation that improves interpretability, reduces bias, and enhances data efficiency in reward modeling.

Findings

Achieves state-of-the-art performance on three benchmarks.

Effectively mitigates evaluation biases like verbosity and position bias.

Requires only 3k high-quality samples for training.

Abstract

Reward modeling is essential for aligning Large Language Models(LLMs) with human preferences, yet conventional reward models suffer from poor interpretability and heavy reliance on costly expert annotations. While recent rubric-based approaches enhance evaluation transparency, they lack systematic quality control, yielding noisy and redundant criteria, failing to mitigate persistent biases (e.g., verbosity, position) in LLM evaluators, and creating a scalability-reliability trade-off. To address these limitations, we propose CDRRM (Contrast-Driven Rubric Reward Model), a framework built on a novel Contrast-then-Synthesis paradigm for high-quality rubric generation and guided preference judgment. CDRRM first conducts multi-dimensional contrastive profiling on preference pairs to identify causal discriminative factors, then synthesizes these insights into compact, context-aware rubrics to guide preference judg- ments. Extensive experiments on three authoritative benchmarks (RewardBench, RMBench, RMB) demonstrate that CDRRM achieves state-of-the-art performance across diverse domains and effectively mitigates aforementioned evaluation biases. Notably, our approach delivers exceptional data efficiency: training the rubric generator on only 3k high-quality samples empowers a frozen pre-trained judge model to outperform fully fine-tuned baselines. This work offers a scalable, interpretable, and data-efficient path for reward modeling.