Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives

TL;DR

This paper critically examines the use of Bradley-Terry models in reward modeling for LLM alignment, providing theoretical foundations, highlighting limitations, and proposing an alternative approach based on order consistency, supported by extensive empirical evaluation.

Contribution

The paper revisits the theoretical basis of BT models in reward modeling, introduces an order-preserving alternative, and empirically compares multiple methods across diverse settings.

Findings

BT models have a solid theoretical foundation but are not necessary for effective reward modeling.

An order-consistent alternative can match or outperform BT models in practice.

Extensive experiments demonstrate the practical viability of the proposed approach across various datasets and models.

Abstract

The Bradley-Terry (BT) model is a common and successful practice in reward modeling for Large Language Model (LLM) alignment. However, it remains unclear why this model -- originally developed for multi-player stochastic game matching -- can be adopted to convert pairwise response comparisons to reward values and make predictions. Especially given the fact that only a limited number of prompt-response pairs are sparsely compared with others. In this paper, we first revisit the foundations of using BT models in reward modeling, and establish the convergence rate of BT reward models based on deep neural networks using embeddings, providing a theoretical foundation for their use. Despite theoretically sound, we argue that the BT model is not a necessary choice from the perspective of downstream optimization. This is because a reward model only needs to preserve the correct ranking predictions through a monotonic transformation of the true reward. We highlight the critical concept of order consistency in reward modeling and demonstrate that the BT model possesses this property. Consequently, we propose a simple and straightforward upper-bound algorithm, compatible with off-the-shelf binary classifiers, as an alternative order-consistent reward modeling objective. To offer practical insights, we empirically evaluate the performance of these different reward modeling approaches across more than 12,000 experimental setups, using $6$ base LLMs, $2$ datasets, and diverse annotation designs that vary in quantity, quality, and pairing choices in preference annotations.