Preference Learning Algorithms Do Not Learn Preference Rankings

TL;DR

This paper reveals that preference learning algorithms like RLHF and DPO do not effectively learn preference rankings, as evidenced by low ranking accuracy and a significant gap from the idealized performance, questioning their core mechanism.

Contribution

The study critically examines the assumption that preference learning models optimize for ranking accuracy, providing theoretical insights and empirical evidence of their limitations and the alignment gap.

Findings

Most preference-tuned models achieve less than 60% ranking accuracy.

Existing models show a significant gap from the idealized ranking accuracy.

Ranking accuracy correlates with win rate when models are close to the reference.

Abstract

Preference learning algorithms (e.g., RLHF and DPO) are frequently used to steer LLMs to produce generations that are more preferred by humans, but our understanding of their inner workings is still limited. In this work, we study the conventional wisdom that preference learning trains models to assign higher likelihoods to more preferred outputs than less preferred outputs, measured via ranking accuracy. Surprisingly, we find that most state-of-the-art preference-tuned models achieve a ranking accuracy of less than 60% on common preference datasets. We furthermore derive the idealized ranking accuracy that a preference-tuned LLM would achieve if it optimized the DPO or RLHF objective perfectly. We demonstrate that existing models exhibit a significant alignment gap -- i.e., a gap between the observed and idealized ranking accuracies. We attribute this discrepancy to the DPO objective, which is empirically and theoretically ill-suited to fix even mild ranking errors in the reference model, and derive a simple and efficient formula for quantifying the difficulty of learning a given preference datapoint. Finally, we demonstrate that ranking accuracy strongly correlates with the empirically popular win rate metric when the model is close to the reference model used in the objective, shedding further light on the differences between on-policy (e.g., RLHF) and off-policy (e.g., DPO) preference learning algorithms.