Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification

TL;DR

This paper studies corruption-robust linear bandit algorithms, characterizing minimax regret bounds under different corruption models, and introduces optimal algorithms for gap-dependent misspecification, advancing understanding in robust learning and reinforcement learning.

Contribution

It provides a unified analysis of corruption types in linear bandits, characterizes the regret gap, and develops optimal algorithms for gap-dependent misspecification, extending to reinforcement learning settings.

Findings

Unified framework for strong and weak corruption models.

Full characterization of minimax regret gap in stochastic linear bandits.

Optimal algorithms for gap-dependent misspecification in linear bandits.

Abstract

In linear bandits, how can a learner effectively learn when facing corrupted rewards? While significant work has explored this question, a holistic understanding across different adversarial models and corruption measures is lacking, as is a full characterization of the minimax regret bounds. In this work, we compare two types of corruptions commonly considered: strong corruption, where the corruption level depends on the action chosen by the learner, and weak corruption, where the corruption level does not depend on the action chosen by the learner. We provide a unified framework to analyze these corruptions. For stochastic linear bandits, we fully characterize the gap between the minimax regret under strong and weak corruptions. We also initiate the study of corrupted adversarial linear bandits, obtaining upper and lower bounds with matching dependencies on the corruption level. Next, we reveal a connection between corruption-robust learning and learning with gap-dependent mis-specification, a setting first studied by Liu et al. (2023a), where the misspecification level of an action or policy is proportional to its suboptimality. We present a general reduction that enables any corruption-robust algorithm to handle gap-dependent misspecification. This allows us to recover the results of Liu et al. (2023a) in a black-box manner and significantly generalize them to settings like linear MDPs, yielding the first results for gap-dependent misspecification in reinforcement learning. However, this general reduction does not attain the optimal rate for gap-dependent misspecification. Motivated by this, we develop a specialized algorithm that achieves optimal bounds for gap-dependent misspecification in linear bandits, thus answering an open question posed by Liu et al. (2023a).