When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models

TL;DR

This paper identifies a bias in the BT-loss used for reward modeling in LLMs caused by representation distance, and proposes NormBT, a normalization method that improves reward model performance by mitigating this bias.

Contribution

The authors analyze the gradient behavior of BT-loss, reveal the impact of representation distance bias, and introduce NormBT, a simple normalization scheme that enhances reward model training.

Findings

NormBT improves reward model performance by over 5% on RewardBench Reasoning.

Representation distance significantly influences gradient magnitude and learning signals.

NormBT consistently outperforms standard BT-loss across various models and datasets.

Abstract

Reward models are central to Large Language Model (LLM) alignment within the framework of RLHF. The standard objective used in reward modeling is the Bradley-Terry (BT) loss, which learns from pairwise data consisting of chosen and rejected responses. In this work, we analyze the per-sample gradient of BT-loss and show spurious learning signals due to representation distance. In particular, BT gradient norm scales with two distinct components: (1) prediction error, reflected by the difference in predicted rewards between chosen and rejected responses, and critically, (2) representation distance between the pair measured in the output space of the final layer. While the first term captures the intended training signal, the second term can significantly impact the update magnitude and misalign learning. Specifically, pairs with small representation distance often receive vanishingly weak updates, even when misranked, while pairs with large distance receive disproportionately strong updates. This leads to gradients from large-distance pairs to overshadow those from small-distance pairs, where fine-grained distinctions are especially important. To overcome this limitation, we propose NormBT, an adaptive pair-wise normalization scheme that rescales updates to balance representation-driven effects and focuses learning signals on prediction error. NormBT is a lightweight, drop-in modification to BT loss with negligible overhead. Across various LLM backbones and datasets, NormBT improves reward model performance consistently, with notable gains of over 5% on the Reasoning category of RewardBench, which contains numerous fine-grained pairs.