Mirror Descent Policy Optimization

TL;DR

This paper introduces Mirror Descent Policy Optimization (MDPO), an efficient RL algorithm inspired by mirror descent, which unifies and improves upon existing trust-region methods like TRPO, PPO, and SAC in continuous control tasks.

Contribution

The paper proposes MDPO, a novel RL algorithm based on mirror descent principles, with on-policy and off-policy variants, connecting and enhancing existing trust-region algorithms.

Findings

MDPO performs better than or comparable to TRPO, PPO, and SAC.

Explicit trust-region constraints are not essential for high performance.

MDPO unifies several popular RL algorithms under a common framework.

Abstract

Mirror descent (MD), a well-known first-order method in constrained convex optimization, has recently been shown as an important tool to analyze trust-region algorithms in reinforcement learning (RL). However, there remains a considerable gap between such theoretically analyzed algorithms and the ones used in practice. Inspired by this, we propose an efficient RL algorithm, called {\em mirror descent policy optimization} (MDPO). MDPO iteratively updates the policy by {\em approximately} solving a trust-region problem, whose objective function consists of two terms: a linearization of the standard RL objective and a proximity term that restricts two consecutive policies to be close to each other. Each update performs this approximation by taking multiple gradient steps on this objective function. We derive {\em on-policy} and {\em off-policy} variants of MDPO, while emphasizing important design choices motivated by the existing theory of MD in RL. We highlight the connections between on-policy MDPO and two popular trust-region RL algorithms: TRPO and PPO, and show that explicitly enforcing the trust-region constraint is in fact {\em not} a necessity for high performance gains in TRPO. We then show how the popular soft actor-critic (SAC) algorithm can be derived by slight modifications of off-policy MDPO. Overall, MDPO is derived from the MD principles, offers a unified approach to viewing a number of popular RL algorithms, and performs better than or on-par with TRPO, PPO, and SAC in a number of continuous control tasks. Code is available at \url{https://github.com/manantomar/Mirror-Descent-Policy-Optimization}.