Generalizing while preserving monotonicity in comparison-based preference learning models

TL;DR

This paper introduces a new class of preference learning models that are both monotone and capable of generalizing to unseen data, addressing a key limitation of existing models like the Generalized Bradley-Terry models.

Contribution

The paper proposes Linear Generalized Bradley-Terry models with Diffusion Priors, providing conditions for monotonicity and enhancing generalization in preference learning.

Findings

The new models guarantee monotonicity under certain conditions.

They outperform existing models in accuracy with limited data.

Monotonicity is not a universal property in preference models.

Abstract

If you tell a learning model that you prefer an alternative $a$ over another alternative $b$, then you probably expect the model to be monotone, that is, the valuation of $a$ increases, and that of $b$ decreases. Yet, perhaps surprisingly, many widely deployed comparison-based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley-Terry models. Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that are monotone. Namely, we propose a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantee monotonicity. Our experiments show that this monotonicity is far from being a general guarantee, and that our new class of generalizing models improves accuracy, especially when the dataset is limited.