Reparameterized Policy Learning for Multimodal Trajectory Optimization

TL;DR

This paper introduces a new multimodal policy parameterization for reinforcement learning that models policies as generative trajectory models, improving exploration and performance in complex, high-dimensional environments.

Contribution

It proposes a novel variational framework for multimodal policy modeling and the RPG algorithm that enhances exploration and data efficiency in RL.

Findings

Outperforms previous methods on various tasks.

Effectively escapes local optima in dense reward environments.

Successfully tackles sparse-reward challenges with intrinsic motivation.

Abstract

We investigate the challenge of parametrizing policies for reinforcement learning (RL) in high-dimensional continuous action spaces. Our objective is to develop a multimodal policy that overcomes limitations inherent in the commonly-used Gaussian parameterization. To achieve this, we propose a principled framework that models the continuous RL policy as a generative model of optimal trajectories. By conditioning the policy on a latent variable, we derive a novel variational bound as the optimization objective, which promotes exploration of the environment. We then present a practical model-based RL method, called Reparameterized Policy Gradient (RPG), which leverages the multimodal policy parameterization and learned world model to achieve strong exploration capabilities and high data efficiency. Empirical results demonstrate that our method can help agents evade local optima in tasks with dense rewards and solve challenging sparse-reward environments by incorporating an object-centric intrinsic reward. Our method consistently outperforms previous approaches across a range of tasks. Code and supplementary materials are available on the project page https://haosulab.github.io/RPG/