CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

TL;DR

CARMO introduces a dynamic, context-aware reward modeling approach that adaptively generates evaluation criteria to improve alignment and reduce reward hacking in large language models, achieving state-of-the-art results.

Contribution

This paper presents CARMO, a novel method that uses large language models to generate context-specific evaluation criteria, enhancing reward modeling and alignment in language models.

Findings

Achieves 2.1% improvement on Reward Bench in zero-shot settings.

Distilled CARMO models reduce computational costs.

Improves LC-WR and WR scores on Mistral-Base (7B).

Abstract

Reward modeling in large language models is susceptible to reward hacking, causing models to latch onto superficial features such as the tendency to generate lists or unnecessarily long responses. In reinforcement learning from human feedback (RLHF) and more generally during post-training flawed reward signals often lead to outputs that optimize for these spurious correlates instead of genuine quality or correctness. We propose Context-Aware Reward Modeling (CARMO), a novel approach that first generates dynamic, context-relevant criteria to ground the reward model before producing reward scores. Unlike prior methods that rely on static rubrics, CARMO leverages large language models (LLMs) to adaptively create evaluation criteria such as logical consistency, clarity, and depth tailored to the user query. Our theoretical analysis shows that such criteria generation can mitigate reward hacking. We further demonstrate that CARMO can be distilled into smaller models, reducing the computational cost of alignment. We establish a new state-of-the-art performance in zero-shot settings for generative models, achieving a 2.1\% improvement on Reward Bench. Furthermore, alignment performed on the CARMO-curated preference dataset achieves 22.5\% and 21.1\% LC-WR and WR, respectively, on Mistral-Base (7B).