Best of Both Worlds Policy Optimization

TL;DR

This paper demonstrates that with proper regularization, policy optimization in tabular MDPs can achieve near-optimal polylogarithmic regret in stochastic settings while maintaining worst-case guarantees in adversarial scenarios.

Contribution

It introduces a novel regularizer design for policy optimization that achieves polylogarithmic regret in stochastic MDPs and first-order regret bounds in adversarial settings.

Findings

Polylog$(T)$ regret in stochastic MDPs with proper regularization.

First first-order regret bound in adversarial MDPs under known transitions.

Regularizer choices include Tsallis and Shannon entropy, and log-barrier.

Abstract

Policy optimization methods are popular reinforcement learning algorithms in practice. Recent works have built theoretical foundation for them by proving $\sqrt{T}$ regret bounds even when the losses are adversarial. Such bounds are tight in the worst case but often overly pessimistic. In this work, we show that in tabular Markov decision processes (MDPs), by properly designing the regularizer, the exploration bonus and the learning rates, one can achieve a more favorable polylog$(T)$ regret when the losses are stochastic, without sacrificing the worst-case guarantee in the adversarial regime. To our knowledge, this is also the first time a gap-dependent polylog$(T)$ regret bound is shown for policy optimization. Specifically, we achieve this by leveraging a Tsallis entropy or a Shannon entropy regularizer in the policy update. Then we show that under known transitions, we can further obtain a first-order regret bound in the adversarial regime by leveraging the log-barrier regularizer.