Online Clustering of Dueling Bandits

TL;DR

This paper introduces novel algorithms for collaborative decision-making in dueling bandit problems using preference feedback, with theoretical guarantees and empirical validation for improved user clustering and regret bounds.

Contribution

It presents the first clustering algorithms for dueling bandits that operate with preference feedback, including linear and neural models, with theoretical analysis and empirical results.

Findings

Algorithms outperform non-collaborative methods in synthetic datasets.

Theoretical regret bounds demonstrate improved performance with user collaboration.

Empirical results on real-world data validate the effectiveness of the proposed methods.

Abstract

The contextual multi-armed bandit (MAB) is a widely used framework for problems requiring sequential decision-making under uncertainty, such as recommendation systems. In applications involving a large number of users, the performance of contextual MAB can be significantly improved by facilitating collaboration among multiple users. This has been achieved by the clustering of bandits (CB) methods, which adaptively group the users into different clusters and achieve collaboration by allowing the users in the same cluster to share data. However, classical CB algorithms typically rely on numerical reward feedback, which may not be practical in certain real-world applications. For instance, in recommendation systems, it is more realistic and reliable to solicit preference feedback between pairs of recommended items rather than absolute rewards. To address this limitation, we introduce the first "clustering of dueling bandit algorithms" to enable collaborative decision-making based on preference feedback. We propose two novel algorithms: (1) Clustering of Linear Dueling Bandits (COLDB) which models the user reward functions as linear functions of the context vectors, and (2) Clustering of Neural Dueling Bandits (CONDB) which uses a neural network to model complex, non-linear user reward functions. Both algorithms are supported by rigorous theoretical analyses, demonstrating that user collaboration leads to improved regret bounds. Extensive empirical evaluations on synthetic and real-world datasets further validate the effectiveness of our methods, establishing their potential in real-world applications involving multiple users with preference-based feedback.