Variational Model-based Policy Optimization

TL;DR

This paper introduces VMBPO, a variational model-based policy optimization method that combines model learning and policy improvement using a unified objective, leading to more sample-efficient and robust reinforcement learning algorithms.

Contribution

It formulates a joint model and policy learning framework via a variational lower-bound, enabling iterative EM-based optimization in model-based RL.

Findings

VMBPO outperforms model-free methods in sample efficiency.

VMBPO demonstrates robustness to hyper-parameter tuning.

VMBPO achieves competitive performance with state-of-the-art algorithms.

Abstract

Model-based reinforcement learning (RL) algorithms allow us to combine model-generated data with those collected from interaction with the real system in order to alleviate the data efficiency problem in RL. However, designing such algorithms is often challenging because the bias in simulated data may overshadow the ease of data generation. A potential solution to this challenge is to jointly learn and improve model and policy using a universal objective function. In this paper, we leverage the connection between RL and probabilistic inference, and formulate such an objective function as a variational lower-bound of a log-likelihood. This allows us to use expectation maximization (EM) and iteratively fix a baseline policy and learn a variational distribution, consisting of a model and a policy (E-step), followed by improving the baseline policy given the learned variational distribution (M-step). We propose model-based and model-free policy iteration (actor-critic) style algorithms for the E-step and show how the variational distribution learned by them can be used to optimize the M-step in a fully model-based fashion. Our experiments on a number of continuous control tasks show that despite being more complex, our model-based (E-step) algorithm, called {\em variational model-based policy optimization} (VMBPO), is more sample-efficient and robust to hyper-parameter tuning than its model-free (E-step) counterpart. Using the same control tasks, we also compare VMBPO with several state-of-the-art model-based and model-free RL algorithms and show its sample efficiency and performance.