Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

TL;DR

This paper introduces a new preference learning framework that aligns policies proportionally with true population preferences, addressing biases and manipulation issues in existing methods, and is validated on recommendation and language model tasks.

Contribution

The paper develops an axiomatic, population-proportional preference aggregation framework that infers evaluator distributions and constructs aligned policies, incorporating novel axioms and a soft-max relaxation method.

Findings

Effective in aligning policies with true population preferences

Scalable to large language models and recommendation tasks

Reduces bias and manipulation in preference aggregation

Abstract

Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation. To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs a policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well as our newly-introduced axioms of population-proportional alignment and population-bounded manipulability. Moreover, we propose a soft-max relaxation method that smoothly trades off population-proportional alignment with the selection of the Condorcet winner (which beats all other options in pairwise comparisons). Finally, we validate the effectiveness and scalability of our approach through experiments on both tabular recommendation tasks and large language model alignment.