Offline Model-Based Reinforcement Learning with Anti-Exploration

TL;DR

This paper introduces MoMo, a novel offline model-based reinforcement learning method that uses anti-exploration techniques to improve performance and stability without relying on large ensembles.

Contribution

MoMo extends anti-exploration to model-based RL, enabling effective out-of-distribution detection and uncertainty handling with fewer models.

Findings

MoMo outperforms prior methods on D4RL datasets.

Model-based MoMo achieves superior results compared to model-free variants.

Anti-exploration improves stability and performance in offline MBRL.

Abstract

Model-based reinforcement learning (MBRL) algorithms learn a dynamics model from collected data and apply it to generate synthetic trajectories to enable faster learning. This is an especially promising paradigm in offline reinforcement learning (RL) where data may be limited in quantity, in addition to being deficient in coverage and quality. Practical approaches to offline MBRL usually rely on ensembles of dynamics models to prevent exploitation of any individual model and to extract uncertainty estimates that penalize values in states far from the dataset support. Uncertainty estimates from ensembles can vary greatly in scale, making it challenging to generalize hyperparameters well across even similar tasks. In this paper, we present Morse Model-based offline RL (MoMo), which extends the anti-exploration paradigm found in offline model-free RL to the model-based space. We develop model-free and model-based variants of MoMo and show how the model-free version can be extended to detect and deal with out-of-distribution (OOD) states using explicit uncertainty estimation without the need for large ensembles. MoMo performs offline MBRL using an anti-exploration bonus to counteract value overestimation in combination with a policy constraint, as well as a truncation function to terminate synthetic rollouts that are excessively OOD. Experimentally, we find that both model-free and model-based MoMo perform well, and the latter outperforms prior model-based and model-free baselines on the majority of D4RL datasets tested.