Sample Efficient Preference Alignment in LLMs via Active Exploration

TL;DR

This paper introduces an active exploration method for preference alignment in large language models, reducing human feedback costs through a formal dueling bandit framework with proven regret bounds.

Contribution

It formalizes preference alignment as an active contextual dueling bandit problem and proposes an efficient algorithm with theoretical guarantees, extending it for practical LLM use.

Findings

Outperforms baselines with limited human preference samples

Effective on multiple language models and datasets

Contributes two new real-world datasets

Abstract

Preference-based feedback is important for many applications in machine learning where evaluation of a reward function is not feasible. Notable recent examples arise in preference alignment for large language models, including in reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO). For many applications of preference alignment, the cost of acquiring human feedback can be substantial. In this work, we take advantage of the fact that one can often choose contexts at which to obtain human feedback to most efficiently identify a good policy, and formalize the setting as an active contextual dueling bandit problem. We propose an active exploration algorithm to efficiently select the data and provide theoretical proof that it has a polynomial worst-case regret bound. We extend the setting and methodology for practical use in preference alignment of large language models. We provide two extensions, an online and an offline approach. Our method outperforms the baselines with limited samples of human preferences on several language models and four real-world datasets including two new datasets that we contribute to the literature.