DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning

TL;DR

This paper introduces DOMAIN, a model-based offline RL algorithm that avoids unreliable uncertainty estimation by adaptively sampling model data, leading to improved performance and safety guarantees.

Contribution

DOMAIN is the first offline RL method that adaptively adjusts model data penalties without relying on uncertainty estimation, enhancing safety and performance.

Findings

DOMAIN outperforms prior algorithms on D4RL benchmark

Theoretical guarantees show lower bound of true Q value outside the region

Performance improved by 1.8% on average

Abstract

Model-based reinforcement learning (RL), which learns an environment model from the offline dataset and generates more out-of-distribution model data, has become an effective approach to the problem of distribution shift in offline RL. Due to the gap between the learned and actual environment, conservatism should be incorporated into the algorithm to balance accurate offline data and imprecise model data. The conservatism of current algorithms mostly relies on model uncertainty estimation. However, uncertainty estimation is unreliable and leads to poor performance in certain scenarios, and the previous methods ignore differences between the model data, which brings great conservatism. To address the above issues, this paper proposes a milDly cOnservative Model-bAsed offlINe RL algorithm (DOMAIN) without estimating model uncertainty, and designs the adaptive sampling distribution of model samples, which can adaptively adjust the model data penalty. In this paper, we theoretically demonstrate that the Q value learned by the DOMAIN outside the region is a lower bound of the true Q value, the DOMAIN is less conservative than previous model-based offline RL algorithms, and has the guarantee of safety policy improvement. The results of extensive experiments show that DOMAIN outperforms prior RL algorithms and the average performance has improved by 1.8% on the D4RL benchmark.