Multi-User Dueling Bandits: A Fair Approach using Nash Social Welfare

TL;DR

This paper introduces a fair dueling bandits framework using Nash Social Welfare to ensure equitable learning across heterogeneous user preferences, providing theoretical bounds and algorithms.

Contribution

It formulates a fairness-aware dueling bandits model with user-specific Condorcet winners and establishes regret bounds, pioneering the quantification of fairness costs in this setting.

Findings

Established a regret lower bound of Ω(T^{2/3} min(K,D)^{1/3}) for fair dueling bandits.

Proposed algorithms with regret bounds matching the lower bound up to logarithmic factors.

Quantified the cost of fairness in heterogeneous preference scenarios.

Abstract

Learning from human preference data is becoming a useful tool, from fine-tuning large language models to training reinforcement learning agents. However, in most scenarios, the model is trained on the average preference of all human evaluators, which, under large variations of preferences, can be unfair to minority groups. In this work, we consider fairness in dueling bandits, a standard framework for online learning from preference data. We assume that each user has a (potentially distinct) Condorcet winner, which is an arm preferred to every other arm. Using these user-specific Condorcet winners as reference points, we evaluate and score arms according to their performance relative to the corresponding winner. To promote fairness across heterogeneous users, we adopt the well-established Nash Social Welfare objective, which maximizes the product of user utilities, thereby inherently penalizing inequality and preventing the marginalization of any single user. Within this framework, we construct a hard instance to establish a regret lower bound of $\Omega(T^{2/3}\min(K,D)^\frac{1}{3})$ for a time horizon $T$, $K$ arms, and $D$ users, which, to the best of our knowledge, is the first result quantifying the cost of fairness in dueling bandits with heterogeneous preferences. We then present the Fair-Explore-Then-Commit and Fair-$\epsilon$-Greedy algorithms with a Condorcet winner identification phase. We further derive their regret upper bounds that match the lower-bound dependence on $T$ up to logarithmic factors.