Model Selection for Generic Contextual Bandits

TL;DR

This paper introduces an adaptive algorithm for model selection in stochastic contextual bandits that automatically adjusts to the true model class, achieving near-optimal regret without prior knowledge of the class.

Contribution

The paper proposes the Adaptive Contextual Bandit (ACB) algorithm that adaptively eliminates overly simple models, matching the regret of algorithms with known model classes, and provides specialized algorithms for linear cases.

Findings

ACB achieves regret comparable to algorithms with known model class.

A simpler explore-then-commit algorithm also attains similar regret bounds.

Specialized algorithms for linear bandits offer sharper guarantees.

Abstract

We consider the problem of model selection for the general stochastic contextual bandits under the realizability assumption. We propose a successive refinement based algorithm called Adaptive Contextual Bandit ({\ttfamily ACB}), that works in phases and successively eliminates model classes that are too simple to fit the given instance. We prove that this algorithm is adaptive, i.e., the regret rate order-wise matches that of any provable contextual bandit algorithm (ex. \cite{falcon}), that needs the knowledge of the true model class. The price of not knowing the correct model class turns out to be only an additive term contributing to the second order term in the regret bound. This cost possess the intuitive property that it becomes smaller as the model class becomes easier to identify, and vice-versa. We also show that a much simpler explore-then-commit (ETC) style algorithm also obtains similar regret bound, despite not knowing the true model class. However, the cost of model selection is higher in ETC as opposed to in {\ttfamily ACB}, as expected. Furthermore, for the special case of linear contextual bandits, we propose specialized algorithms that obtain sharper guarantees compared to the generic setup.