Kernel $\epsilon$-Greedy for Multi-Armed Bandits with Covariates

TL;DR

This paper introduces a kernel-based $ ext{epsilon}$-greedy method for multi-armed bandits with covariates, achieving optimal regret rates by leveraging RKHS properties and online kernel ridge regression.

Contribution

It proposes a novel kernel $ ext{epsilon}$-greedy algorithm with theoretical guarantees for consistency and regret minimization in MABC problems.

Findings

Achieves sub-linear regret depending on RKHS dimensionality.

Attains the optimal $ ext{sqrt}(T)$ regret under certain conditions.

Provides a consistent estimator for mean reward functions.

Abstract

We consider the $\epsilon$-greedy strategy for the multi-arm bandit with covariates (MABC) problem, where the mean reward functions are assumed to lie in a reproducing kernel Hilbert space (RKHS). We propose to estimate the unknown mean reward functions using an online weighted kernel ridge regression estimator, and show the resultant estimator to be consistent under appropriate decay rates of the exploration probability sequence, $\{\epsilon_t\}_t$, and regularization parameter, $\{\lambda_t\}_t$. Moreover, we show that for any choice of kernel and the corresponding RKHS, we achieve a sub-linear regret rate depending on the intrinsic dimensionality of the RKHS. Furthermore, we achieve the optimal regret rate of $\sqrt{T}$ under a margin condition for finite-dimensional RKHS.