The best of both worlds: stochastic and adversarial episodic MDPs with unknown transition

TL;DR

This paper develops a new algorithmic approach for episodic MDPs that achieves near-optimal regret bounds in both stochastic and adversarial settings, even when the transition dynamics are unknown.

Contribution

It introduces a loss-shifting technique within the FTRL framework to handle unknown transitions, resolving a major open problem in the field.

Findings

Achieves rac{ ext{polylog}(T)}{ ext{regret}} in stochastic setting

Attains rac{ ext{ extasciitilde}\

rac{ ext{ extasciitilde}\

Abstract

We consider the best-of-both-worlds problem for learning an episodic Markov Decision Process through $T$ episodes, with the goal of achieving $\widetilde{\mathcal{O}}(\sqrt{T})$ regret when the losses are adversarial and simultaneously $\mathcal{O}(\text{polylog}(T))$ regret when the losses are (almost) stochastic. Recent work by [Jin and Luo, 2020] achieves this goal when the fixed transition is known, and leaves the case of unknown transition as a major open question. In this work, we resolve this open problem by using the same Follow-the-Regularized-Leader ($\text{FTRL}$) framework together with a set of new techniques. Specifically, we first propose a loss-shifting trick in the $\text{FTRL}$ analysis, which greatly simplifies the approach of [Jin and Luo, 2020] and already improves their results for the known transition case. Then, we extend this idea to the unknown transition case and develop a novel analysis which upper bounds the transition estimation error by (a fraction of) the regret itself in the stochastic setting, a key property to ensure $\mathcal{O}(\text{polylog}(T))$ regret.