Comparison-based Active Preference Learning for Multi-dimensional Personalization

TL;DR

This paper introduces AMPLe, an active learning framework that efficiently captures implicit multi-dimensional user preferences for personalized language model responses, reducing the number of comparisons needed.

Contribution

It presents a novel active preference learning method with a modified Bayesian update and query strategy for multi-dimensional personalization of language models.

Findings

Effective in capturing implicit preferences

Reduces number of comparisons needed

Improves personalization accuracy

Abstract

Large language models (LLMs) have shown remarkable success, but aligning them with human preferences remains a core challenge. As individuals have their own, multi-dimensional preferences, recent studies have explored multi-dimensional personalization, which aims to enable models to generate responses personalized to explicit preferences. However, human preferences are often implicit and thus difficult to articulate, limiting the direct application of this approach. To bridge this gap, we propose Active Multi-dimensional Preference Learning (AMPLe), designed to capture implicit user preferences from interactively collected comparative feedback. Building on Bayesian inference, our work introduces a modified posterior update procedure to mitigate estimation bias and potential noise in comparisons. Also, inspired by generalized binary search, we employ an active query selection strategy to minimize the number of required comparisons by a user. Through theoretical analysis and experiments on language generation tasks, we demonstrate feedback efficiency and effectiveness of our framework in personalizing model responses. Our code is publicly available at https://github.com/ml-postech/AMPLe .