Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisited

TL;DR

This paper establishes new fundamental lower bounds on sample complexity and regret for episodic non-stationary MDPs, advancing understanding of the limits of reinforcement learning in changing environments.

Contribution

It introduces novel problem-independent lower bounds for sample complexity and regret in non-stationary episodic MDPs, using new constructions of hard MDPs.

Findings

Lower bound of Ω((H^3SA/ε^2) log(1/δ)) on sample complexity for PAC algorithms.

Regret lower bound of Ω(√(H^3SAT)) for non-stationary MDPs.

Connections to PAC-MDP lower bounds are discussed.

Abstract

In this paper, we propose new problem-independent lower bounds on the sample complexity and regret in episodic MDPs, with a particular focus on the non-stationary case in which the transition kernel is allowed to change in each stage of the episode. Our main contribution is a novel lower bound of $\Omega((H^3SA/\epsilon^2)\log(1/\delta))$ on the sample complexity of an $(\varepsilon,\delta)$-PAC algorithm for best policy identification in a non-stationary MDP. This lower bound relies on a construction of "hard MDPs" which is different from the ones previously used in the literature. Using this same class of MDPs, we also provide a rigorous proof of the $\Omega(\sqrt{H^3SAT})$ regret bound for non-stationary MDPs. Finally, we discuss connections to PAC-MDP lower bounds.