Nonparametric Stochastic Contextual Bandits

TL;DR

This paper introduces a nonparametric approach to contextual bandits using a simple kNN-UCB algorithm, achieving tight theoretical bounds and demonstrating superior performance in experiments including image classification.

Contribution

It presents a novel nonparametric bandit algorithm with tight theoretical guarantees and practical improvements over existing methods, extending to infinite-armed settings.

Findings

Achieves sublinear regret of T^{(1+D)/(2+D)} with context dimension D.

Provides dimension-dependent and independent regret bounds.

Demonstrates improved performance on simulated tasks and MNIST classification.

Abstract

We analyze the $K$-armed bandit problem where the reward for each arm is a noisy realization based on an observed context under mild nonparametric assumptions. We attain tight results for top-arm identification and a sublinear regret of $\widetilde{O}\Big(T^{\frac{1+D}{2+D}}\Big)$, where $D$ is the context dimension, for a modified UCB algorithm that is simple to implement ($k$NN-UCB). We then give global intrinsic dimension dependent and ambient dimension independent regret bounds. We also discuss recovering topological structures within the context space based on expected bandit performance and provide an extension to infinite-armed contextual bandits. Finally, we experimentally show the improvement of our algorithm over existing multi-armed bandit approaches for both simulated tasks and MNIST image classification.