Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processes

TL;DR

This paper introduces a new computationally efficient algorithm for reinforcement learning in adversarially corrupted environments, achieving near-optimal regret bounds with general function approximation in both contextual bandits and Markov decision processes.

Contribution

It develops a novel uncertainty-weighted least-squares approach for general function classes, extending robust RL algorithms beyond linear settings with improved regret bounds.

Findings

Achieves regret of ten + zeta for corrupted RL.

Generalizes to episodic MDPs with additive corruption dependence.

Outperforms existing methods across all corruption levels.

Abstract

Despite the significant interest and progress in reinforcement learning (RL) problems with adversarial corruption, current works are either confined to the linear setting or lead to an undesired $\tilde{O}(\sqrt{T}\zeta)$ regret bound, where $T$ is the number of rounds and $\zeta$ is the total amount of corruption. In this paper, we consider the contextual bandit with general function approximation and propose a computationally efficient algorithm to achieve a regret of $\tilde{O}(\sqrt{T}+\zeta)$. The proposed algorithm relies on the recently developed uncertainty-weighted least-squares regression from linear contextual bandit and a new weighted estimator of uncertainty for the general function class. In contrast to the existing analysis that heavily relies on the linear structure, we develop a novel technique to control the sum of weighted uncertainty, thus establishing the final regret bounds. We then generalize our algorithm to the episodic MDP setting and first achieve an additive dependence on the corruption level $\zeta$ in the scenario of general function approximation. Notably, our algorithms achieve regret bounds either nearly match the performance lower bound or improve the existing methods for all the corruption levels and in both known and unknown $\zeta$ cases.