Variance-Aware Linear UCB with Deep Representation for Neural Contextual Bandits

TL;DR

This paper introduces a variance-aware neural UCB algorithm that leverages deep neural networks and variance estimation to improve exploration, leading to better regret bounds and empirical performance in neural contextual bandits.

Contribution

It proposes a novel variance-aware neural UCB algorithm with theoretical regret guarantees and practical variance estimation, outperforming existing methods.

Findings

Achieves better regret bounds than previous neural-UCB algorithms.

Outperforms state-of-the-art methods on multiple datasets.

Provides a practical variance estimation method with similar computational efficiency.

Abstract

By leveraging the representation power of deep neural networks, neural upper confidence bound (UCB) algorithms have shown success in contextual bandits. To further balance the exploration and exploitation, we propose Neural-$\sigma^2$-LinearUCB, a variance-aware algorithm that utilizes $\sigma^2_t$, i.e., an upper bound of the reward noise variance at round $t$, to enhance the uncertainty quantification quality of the UCB, resulting in a regret performance improvement. We provide an oracle version for our algorithm characterized by an oracle variance upper bound $\sigma^2_t$ and a practical version with a novel estimation for this variance bound. Theoretically, we provide rigorous regret analysis for both versions and prove that our oracle algorithm achieves a better regret guarantee than other neural-UCB algorithms in the neural contextual bandits setting. Empirically, our practical method enjoys a similar computational efficiency, while outperforming state-of-the-art techniques by having a better calibration and lower regret across multiple standard settings, including on the synthetic, UCI, MNIST, and CIFAR-10 datasets.