Online Learning in MDPs with Partially Adversarial Transitions and Losses

TL;DR

This paper introduces algorithms for reinforcement learning in Markov Decision Processes with mostly stochastic but occasionally adversarial transitions, providing regret bounds and characterizing the difficulty of learning under such conditions.

Contribution

It proposes the concept of conditioned occupancy measures and develops algorithms with regret bounds for MDPs with partially adversarial transitions, including cases with consecutive adversarial steps and unknown adversarial points.

Findings

Achieves regret of ;O(H S^{\u00A0A0} A0 ext{K} S A^{A0+1}) for arbitrary adversarial steps.

Improves regret dependence to A0;O(H A0 ext{K} S^{3} A^{A0+1}) when adversarial steps are consecutive.

Provides regret bounds for fully adversarial MDPs under different feedback models.

Abstract

We study reinforcement learning in MDPs whose transition function is stochastic at most steps but may behave adversarially at a fixed subset of $\Lambda$ steps per episode. This model captures environments that are stable except at a few vulnerable points. We introduce \emph{conditioned occupancy measures}, which remain stable across episodes even with adversarial transitions, and use them to design two algorithms. The first handles arbitrary adversarial steps and achieves regret $\tilde{O}(H S^{\Lambda}\sqrt{K S A^{\Lambda+1}})$, where $K$ is the number of episodes, $S$ is the number of state, $A$ is the number of actions and $H$ is the episode's horizon. The second, assuming the adversarial steps are consecutive, improves the dependence on $S$ to $\tilde{O}(H\sqrt{K S^{3} A^{\Lambda+1}})$. We further give a $K^{2/3}$-regret reduction that removes the need to know which steps are the $\Lambda$ adversarial steps. We also characterize the regret of adversarial MDPs in the \emph{fully adversarial} setting ($\Lambda=H-1$) both for full-information and bandit feedback, and provide almost matching upper and lower bounds (slightly strengthen existing lower bounds, and clarify how different feedback structures affect the hardness of learning).