Mirror Learning: A Unifying Framework of Policy Optimisation

TL;DR

Mirror Learning introduces a unifying theoretical framework for policy optimization in deep reinforcement learning, explaining the success of scalable algorithms and enabling the development of new, theoretically sound methods.

Contribution

The paper proposes Mirror Learning as a comprehensive framework that encompasses existing algorithms like TRPO and PPO, providing theoretical guarantees and unifying diverse approaches.

Findings

Mirror Learning includes TRPO and PPO as special cases.

It offers theoretical guarantees for a broad class of algorithms.

Empirical success is linked to theoretical properties rather than heuristics.

Abstract

Modern deep reinforcement learning (RL) algorithms are motivated by either the generalised policy iteration (GPI) or trust-region learning (TRL) frameworks. However, algorithms that strictly respect these theoretical frameworks have proven unscalable. Surprisingly, the only known scalable algorithms violate the GPI/TRL assumptions, e.g. due to required regularisation or other heuristics. The current explanation of their empirical success is essentially "by analogy": they are deemed approximate adaptations of theoretically sound methods. Unfortunately, studies have shown that in practice these algorithms differ greatly from their conceptual ancestors. In contrast, in this paper we introduce a novel theoretical framework, named Mirror Learning, which provides theoretical guarantees to a large class of algorithms, including TRPO and PPO. While the latter two exploit the flexibility of our framework, GPI and TRL fit in merely as pathologically restrictive corner cases thereof. This suggests that the empirical performance of state-of-the-art methods is a direct consequence of their theoretical properties, rather than of aforementioned approximate analogies. Mirror learning sets us free to boldly explore novel, theoretically sound RL algorithms, a thus far uncharted wonderland.