Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

TL;DR

This paper introduces a new metric called distortion to evaluate how well AI alignment methods optimize for diverse human preferences, revealing that current methods like RLHF and DPO often perform poorly in pluralistic settings.

Contribution

The paper models user preferences with Bradley-Terry models and establishes a theoretical framework to measure alignment quality via distortion, highlighting limitations of existing methods.

Findings

Nash Learning achieves minimax optimal distortion of approximately half of the BT temperature.

RLHF and DPO exhibit high distortion, often close to the maximum possible, especially without KL constraints.

Distortion varies significantly depending on preference sampling and comparison pair distributions.

Abstract

After pre-training, large language models are aligned with human preferences based on pairwise comparisons. State-of-the-art alignment methods (such as PPO-based RLHF and DPO) are built on the assumption of aligning with a single preference model, despite being deployed in settings where users have diverse preferences. As a result, it is not even clear that these alignment methods produce models that satisfy users on average -- a minimal requirement for pluralistic alignment. Drawing on social choice theory and modeling users' comparisons through individual Bradley-Terry (BT) models, we introduce an alignment method's distortion: the worst-case ratio between the optimal achievable average utility, and the average utility of the learned policy. The notion of distortion helps draw sharp distinctions between alignment methods: Nash Learning from Human Feedback achieves the minimax optimal distortion of $(\frac{1}{2} + o(1)) \cdot \beta$ (for the BT temperature $\beta$), robustly across utility distributions, distributions of comparison pairs, and permissible KL divergences from the reference policy. RLHF and DPO, by contrast, suffer $\geq (1 - o(1)) \cdot \beta$ distortion already without a KL constraint, and $e^{\Omega(\beta)}$ or even unbounded distortion in the full setting, depending on how comparison pairs are sampled.