Model Generation with Provable Coverability for Offline Reinforcement Learning

TL;DR

This paper introduces a provably coverable model generation method for offline reinforcement learning, improving out-of-distribution generalization and transfer performance by generating models that better approximate real dynamics.

Contribution

We propose a novel algorithm for generating models with guaranteed coverage of real dynamics, backed by theoretical analysis and improved empirical performance.

Findings

Outperforms prior offline RL methods on benchmarks

Models exhibit better zero-shot transfer performance

Provides theoretical guarantees on model coverability

Abstract

Model-based offline optimization with dynamics-aware policy provides a new perspective for policy learning and out-of-distribution generalization, where the learned policy could adapt to different dynamics enumerated at the training stage. But due to the limitation under the offline setting, the learned model could not mimic real dynamics well enough to support reliable out-of-distribution exploration, which still hinders policy to generalize well. To narrow the gap, previous works roughly ensemble randomly initialized models to better approximate the real dynamics. However, such practice is costly and inefficient, and provides no guarantee on how well the real dynamics could be approximated by the learned models, which we name coverability in this paper. We actively address this issue by generating models with provable ability to cover real dynamics in an efficient and controllable way. To that end, we design a distance metric for dynamic models based on the occupancy of policies under the dynamics, and propose an algorithm to generate models optimizing their coverage for the real dynamics. We give a theoretical analysis on the model generation process and proves that our algorithm could provide enhanced coverability. As a downstream task, we train a dynamics-aware policy with minor or no conservative penalty, and experiments demonstrate that our algorithm outperforms prior offline methods on existing offline RL benchmarks. We also discover that policies learned by our method have better zero-shot transfer performance, implying their better generalization.