Reducing Dueling Bandits to Cardinal Bandits

TL;DR

This paper introduces algorithms that convert the Dueling Bandits problem into the well-studied stochastic Multi-Armed Bandits problem, enabling the use of existing algorithms and analysis tools for ordinal feedback scenarios.

Contribution

The paper proposes three reduction algorithms, providing regret bounds and empirical performance improvements over previous methods for Dueling Bandits.

Findings

Regret bounds established for two reduction algorithms.

Empirical results show superior performance of the proposed methods.

First almost optimal regret bound considering second order differences.

Abstract

We present algorithms for reducing the Dueling Bandits problem to the conventional (stochastic) Multi-Armed Bandits problem. The Dueling Bandits problem is an online model of learning with ordinal feedback of the form "A is preferred to B" (as opposed to cardinal feedback like "A has value 2.5"), giving it wide applicability in learning from implicit user feedback and revealed and stated preferences. In contrast to existing algorithms for the Dueling Bandits problem, our reductions -- named $\Doubler$, $\MultiSbm$ and $\DoubleSbm$ -- provide a generic schema for translating the extensive body of known results about conventional Multi-Armed Bandit algorithms to the Dueling Bandits setting. For $\Doubler$ and $\MultiSbm$ we prove regret upper bounds in both finite and infinite settings, and conjecture about the performance of $\DoubleSbm$ which empirically outperforms the other two as well as previous algorithms in our experiments. In addition, we provide the first almost optimal regret bound in terms of second order terms, such as the differences between the values of the arms.