R3: Robust Rubric-Agnostic Reward Models

TL;DR

R3 introduces a flexible, interpretable reward modeling framework that aligns language models with human preferences across diverse tasks without relying on fixed rubrics.

Contribution

The paper presents R3, a novel rubric-agnostic reward model that enhances interpretability and generalizability in evaluating language models.

Findings

R3 provides transparent, reasoned score assignments.

R3 is adaptable across multiple evaluation dimensions.

Open-source implementation available for broader use.

Abstract

Reward models are essential for aligning language model outputs with human preferences, yet existing approaches often lack both controllability and interpretability. These models are typically optimized for narrow objectives, limiting their generalizability to broader downstream tasks. Moreover, their scalar outputs are difficult to interpret without contextual reasoning. To address these limitations, we introduce $\shortmethodname$, a novel reward modeling framework that is rubric-agnostic, generalizable across evaluation dimensions, and provides interpretable, reasoned score assignments. $\shortmethodname$ enables more transparent and flexible evaluation of language models, supporting robust alignment with diverse human values and use cases. Our models, data, and code are available as open source at https://github.com/rubricreward/r3.