Model-based Policy Optimization with Unsupervised Model Adaptation

TL;DR

This paper introduces AMPO, a model-based reinforcement learning framework that uses unsupervised model adaptation to explicitly reduce the distribution mismatch between real and simulated data, improving policy optimization.

Contribution

The paper proposes a novel framework with unsupervised model adaptation to explicitly align real and simulated data distributions in model-based RL.

Findings

Achieves state-of-the-art sample efficiency on continuous control tasks

Uses Wasserstein-1 distance for practical implementation

Effectively reduces distribution mismatch in model-based RL

Abstract

Model-based reinforcement learning methods learn a dynamics model with real data sampled from the environment and leverage it to generate simulated data to derive an agent. However, due to the potential distribution mismatch between simulated data and real data, this could lead to degraded performance. Despite much effort being devoted to reducing this distribution mismatch, existing methods fail to solve it explicitly. In this paper, we investigate how to bridge the gap between real and simulated data due to inaccurate model estimation for better policy optimization. To begin with, we first derive a lower bound of the expected return, which naturally inspires a bound maximization algorithm by aligning the simulated and real data distributions. To this end, we propose a novel model-based reinforcement learning framework AMPO, which introduces unsupervised model adaptation to minimize the integral probability metric (IPM) between feature distributions from real and simulated data. Instantiating our framework with Wasserstein-1 distance gives a practical model-based approach. Empirically, our approach achieves state-of-the-art performance in terms of sample efficiency on a range of continuous control benchmark tasks.