Catoni Contextual Bandits are Robust to Heavy-tailed Rewards

TL;DR

This paper introduces a robust algorithm for contextual bandits that handles heavy-tailed rewards, achieving regret bounds that depend only on reward variance and logarithmically on the reward range, improving robustness over traditional methods.

Contribution

The paper develops a new algorithm using Catoni's estimator for robust contextual bandits, with regret bounds that are less sensitive to reward range and heavy tails, including unknown variance scenarios.

Findings

Regret depends only on reward variance and logarithmically on reward range R.

Proposed algorithms are robust to heavy-tailed rewards and unknown variances.

Matching lower bounds demonstrate the optimality of the regret bounds.

Abstract

Typical contextual bandit algorithms assume that the rewards at each round lie in some fixed range $[0, R]$, and their regret scales polynomially with this reward range $R$. However, many practical scenarios naturally involve heavy-tailed rewards or rewards where the worst-case range can be substantially larger than the variance. In this paper, we develop an algorithmic approach building on Catoni's estimator from robust statistics, and apply it to contextual bandits with general function approximation. When the variance of the reward at each round is known, we use a variance-weighted regression approach and establish a regret bound that depends only on the cumulative reward variance and logarithmically on the reward range $R$ as well as the number of rounds $T$. For the unknown-variance case, we further propose a careful peeling-based algorithm and remove the need for cumbersome variance estimation. With additional dependence on the fourth moment, our algorithm also enjoys a variance-based bound with logarithmic reward-range dependence. Moreover, we demonstrate the optimality of the leading-order term in our regret bound through a matching lower bound.