Calibrated Model-Based Deep Reinforcement Learning

TL;DR

This paper demonstrates that calibrating predictive uncertainties in deep model-based reinforcement learning enhances planning, exploration, and sample efficiency, leading to state-of-the-art results on the HalfCheetah task with fewer samples.

Contribution

It introduces a simple method to augment any model-based RL agent with calibrated models, significantly improving performance and sample efficiency.

Findings

Calibrated models improve planning and exploration.

Achieved state-of-the-art results on HalfCheetah with 50% fewer samples.

Calibration adds minimal computational overhead.

Abstract

Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties---especially ones derived from modern deep learning systems---can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that good uncertainties must be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the \textsc{HalfCheetah} MuJoCo task, our system achieves state-of-the-art performance using 50\% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.