Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?

TL;DR

This paper compares different generative models for micro-data model-based reinforcement learning, revealing that mixture density networks excel in multimodal environments, while deterministic models perform well otherwise, and introduces evaluation metrics improving sample efficiency.

Contribution

It provides a rigorous comparison of generative models in MBRL, introduces evaluation metrics, and demonstrates significant sample complexity improvements on Acrobot.

Findings

Mixture density nets outperform others in multimodal environments.

Deterministic models perform comparably to probabilistic ones when multimodality isn't needed.

Heteroscedastic training improves long-horizon predictions.

Abstract

We contribute to micro-data model-based reinforcement learning (MBRL) by rigorously comparing popular generative models using a fixed (random shooting) control agent. We find that on an environment that requires multimodal posterior predictives, mixture density nets outperform all other models by a large margin. When multimodality is not required, our surprising finding is that we do not need probabilistic posterior predictives: deterministic models are on par, in fact they consistently (although non-significantly) outperform their probabilistic counterparts. We also found that heteroscedasticity at training time, perhaps acting as a regularizer, improves predictions at longer horizons. At the methodological side, we design metrics and an experimental protocol which can be used to evaluate the various models, predicting their asymptotic performance when using them on the control problem. Using this framework, we improve the state-of-the-art sample complexity of MBRL on Acrobot by two to four folds, using an aggressive training schedule which is outside of the hyperparameter interval usually considered