Robust Reward Modeling for Large Language Models via Causal Decomposition

TL;DR

This paper introduces a causal decomposition approach to improve reward modeling for large language models, reducing overfitting to spurious cues and better aligning outputs with prompt intent.

Contribution

It proposes a decoder-based regularization method that emphasizes prompt-dependent information, enhancing reward model robustness and alignment.

Findings

Decoder achieves 87.7% accuracy in selecting prompt-aligned responses.

Increases RewardBench accuracy from 83.2% to 86.8%.

Produces shorter, less sycophantic outputs while maintaining robustness.

Abstract

Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.