An Asymptotically Optimal Batched Algorithm for the Dueling Bandit Problem

TL;DR

This paper introduces a batched algorithm for the dueling bandit problem that achieves asymptotic optimal regret with only logarithmic rounds, making it suitable for large-scale applications where sequential updates are impractical.

Contribution

The paper presents the first asymptotically optimal batched algorithm for the dueling bandit problem under the Condorcet condition, matching sequential regret bounds with significantly fewer rounds.

Findings

Achieves asymptotic regret of O(K^2 log^2(K)) + O(K log T) in O(log T) rounds.

Performs nearly as well as fully sequential algorithms in real-world datasets.

Uses only logarithmic rounds, reducing the need for frequent updates in large-scale systems.

Abstract

We study the $K$-armed dueling bandit problem, a variation of the traditional multi-armed bandit problem in which feedback is obtained in the form of pairwise comparisons. Previous learning algorithms have focused on the $\textit{fully adaptive}$ setting, where the algorithm can make updates after every comparison. The "batched" dueling bandit problem is motivated by large-scale applications like web search ranking and recommendation systems, where performing sequential updates may be infeasible. In this work, we ask: $\textit{is there a solution using only a few adaptive rounds that matches the asymptotic regret bounds of the best sequential algorithms for $K$-armed dueling bandits?}$ We answer this in the affirmative $\textit{under the Condorcet condition}$, a standard setting of the $K$-armed dueling bandit problem. We obtain asymptotic regret of $O(K^2\log^2(K)) + O(K\log(T))$ in $O(\log(T))$ rounds, where $T$ is the time horizon. Our regret bounds nearly match the best regret bounds known in the fully sequential setting under the Condorcet condition. Finally, in computational experiments over a variety of real-world datasets, we observe that our algorithm using $O(\log(T))$ rounds achieves almost the same performance as fully sequential algorithms (that use $T$ rounds).