Maximum a Posteriori Policy Optimisation

TL;DR

Maximum a Posteriori Policy Optimisation (MPO) is a new reinforcement learning algorithm that improves sample efficiency, robustness, and convergence in continuous control tasks by using a coordinate ascent approach on a relative entropy objective.

Contribution

The paper introduces MPO, a novel off-policy reinforcement learning algorithm based on coordinate ascent on a relative entropy objective, unifying and extending existing methods.

Findings

Outperforms existing methods in sample efficiency

Demonstrates robustness to hyperparameters

Achieves competitive or superior final performance

Abstract

We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.