Reward-Biased Maximum Likelihood Estimation for Neural Contextual Bandits

TL;DR

This paper introduces NeuralRBMLE, a neural network-based approach for stochastic contextual bandits that incorporates reward-biased maximum likelihood estimation to enhance exploration and achieve competitive regret bounds.

Contribution

It adapts the classic RBMLE principle with neural networks for contextual bandits, proposing two algorithms with theoretical regret guarantees and superior empirical performance.

Findings

Both algorithms achieve rac{}{}( ilde{O}(\u221a{T})) regret.

NeuralRBMLE methods outperform state-of-the-art on real datasets.

The approach encodes exploration directly in neural network parameters.

Abstract

Reward-biased maximum likelihood estimation (RBMLE) is a classic principle in the adaptive control literature for tackling explore-exploit trade-offs. This paper studies the stochastic contextual bandit problem with general bounded reward functions and proposes NeuralRBMLE, which adapts the RBMLE principle by adding a bias term to the log-likelihood to enforce exploration. NeuralRBMLE leverages the representation power of neural networks and directly encodes exploratory behavior in the parameter space, without constructing confidence intervals of the estimated rewards. We propose two variants of NeuralRBMLE algorithms: The first variant directly obtains the RBMLE estimator by gradient ascent, and the second variant simplifies RBMLE to a simple index policy through an approximation. We show that both algorithms achieve $\widetilde{\mathcal{O}}(\sqrt{T})$ regret. Through extensive experiments, we demonstrate that the NeuralRBMLE algorithms achieve comparable or better empirical regrets than the state-of-the-art methods on real-world datasets with non-linear reward functions.