Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe

TL;DR

This paper introduces a randomized Frank-Wolfe algorithm for efficiently learning human preferences modeled by Plackett-Luce from limited comparison data, significantly reducing computational complexity.

Contribution

It proposes a novel randomized Frank-Wolfe approach to solve D-optimal design problems for preference learning, enabling scalable and efficient data collection.

Findings

The algorithm effectively reduces computational complexity.

Empirical results demonstrate strong performance on NLP datasets.

The method outperforms traditional approaches in efficiency.

Abstract

We study learning of human preferences from a limited comparison feedback. This task is ubiquitous in machine learning. Its applications such as reinforcement learning from human feedback, have been transformational. We formulate this problem as learning a Plackett-Luce model over a universe of $N$ choices from $K$-way comparison feedback, where typically $K \ll N$. Our solution is the D-optimal design for the Plackett-Luce objective. The design defines a data logging policy that elicits comparison feedback for a small collection of optimally chosen points from all ${N \choose K}$ feasible subsets. The main algorithmic challenge in this work is that even fast methods for solving D-optimal designs would have $O({N \choose K})$ time complexity. To address this issue, we propose a randomized Frank-Wolfe (FW) algorithm that solves the linear maximization sub-problems in the FW method on randomly chosen variables. We analyze the algorithm, and evaluate it empirically on synthetic and open-source NLP datasets.