Neural Dueling Bandits: Preference-Based Optimization with Human Feedback

TL;DR

This paper introduces neural network-based algorithms for preference-based contextual bandit problems, overcoming linear reward assumptions and providing theoretical guarantees with empirical validation.

Contribution

It proposes novel neural dueling bandit algorithms with regret guarantees and extends theoretical analysis to binary feedback scenarios.

Findings

Algorithms achieve sub-linear regret in experiments.

Neural models outperform linear reward models in complex settings.

Theoretical bounds are validated empirically.

Abstract

Contextual dueling bandit is used to model the bandit problems, where a learner's goal is to find the best arm for a given context using observed noisy human preference feedback over the selected arms for the past contexts. However, existing algorithms assume the reward function is linear, which can be complex and non-linear in many real-life applications like online recommendations or ranking web search results. To overcome this challenge, we use a neural network to estimate the reward function using preference feedback for the previously selected arms. We propose upper confidence bound- and Thompson sampling-based algorithms with sub-linear regret guarantees that efficiently select arms in each round. We also extend our theoretical results to contextual bandit problems with binary feedback, which is in itself a non-trivial contribution. Experimental results on the problem instances derived from synthetic datasets corroborate our theoretical results.