Reward Models Inherit Value Biases from Pretraining

TL;DR

This study reveals that reward models inherit and reflect the value biases of their base pretrained language models, affecting their alignment with human values and emphasizing the importance of considering pretraining choices in safety efforts.

Contribution

The paper demonstrates that reward models inherit value biases from their pretrained models, with empirical evidence showing persistent agency and communion preferences across different RMs.

Findings

RMs show significant value biases based on their base models.

Logit differences can be used as implicit reward scores reflecting these biases.

Biases are durable across different training conditions.

Abstract

Reward models (RMs) are central to aligning large language models (LLMs) with human values but have received less attention than pretrained and post-trained LLMs themselves. Because RMs are initialized from LLMs, they inherit representations that shape their behavior, but the nature and extent of this influence remain understudied. In a comprehensive study of 10 leading open-weight RMs using validated psycholinguistic corpora, we show that RMs exhibit significant differences along multiple dimensions of human value as a function of their base model. Using the "Big Two" psychological axes, we show a robust preference of Llama RMs for "agency" and a corresponding robust preference of Gemma RMs for "communion." This phenomenon holds even when the preference data and finetuning process are identical, and we trace it back to the logits of the respective instruction-tuned and pretrained models. These log-probability differences themselves can be formulated as an implicit RM; we derive usable implicit reward scores and show that they exhibit the very same agency/communion difference. We run experiments training RMs with ablations for preference data source and quantity, which demonstrate that this effect is not only repeatable but surprisingly durable. Despite RMs being designed to represent human preferences, our evidence shows that their outputs are influenced by the pretrained LLMs on which they are based. This work underscores the importance of safety and alignment efforts at the pretraining stage, and makes clear that open-source developers' choice of base model is as much a consideration of values as of performance.