Dueling Bandits: From Two-dueling to Multi-dueling

TL;DR

This paper introduces new algorithms for the multi-dueling bandit problem, extending the traditional two-dueling setting to multiple options, with theoretical guarantees and empirical validation showing improved regret bounds and performance.

Contribution

It proposes the DoublerBAI, MultiSBM-Feedback, and MultiRUCB algorithms, providing regret bounds and finite-time analysis for the multi-dueling bandit problem, a significant generalization of the classic two-dueling setting.

Findings

Algorithms achieve $O( ext{ln } T)$ regret bounds.

MultiSBM-Feedback reduces constant factors compared to benchmarks.

Empirical results show outperforming existing algorithms on synthetic and real data.

Abstract

We study a general multi-dueling bandit problem, where an agent compares multiple options simultaneously and aims to minimize the regret due to selecting suboptimal arms. This setting generalizes the traditional two-dueling bandit problem and finds many real-world applications involving subjective feedback on multiple options. We start with the two-dueling bandit setting and propose two efficient algorithms, DoublerBAI and MultiSBM-Feedback. DoublerBAI provides a generic schema for translating known results on best arm identification algorithms to the dueling bandit problem, and achieves a regret bound of $O(\ln T)$. MultiSBM-Feedback not only has an optimal $O(\ln T)$ regret, but also reduces the constant factor by almost a half compared to benchmark results. Then, we consider the general multi-dueling case and develop an efficient algorithm MultiRUCB. Using a novel finite-time regret analysis for the general multi-dueling bandit problem, we show that MultiRUCB also achieves an $O(\ln T)$ regret bound and the bound tightens as the capacity of the comparison set increases. Based on both synthetic and real-world datasets, we empirically demonstrate that our algorithms outperform existing algorithms.