Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief

TL;DR

This paper introduces a novel offline RL method that maintains a belief distribution over dynamics, using pessimism-modulated sampling to improve policy learning and achieve state-of-the-art results.

Contribution

It proposes a new approach of biased sampling from a dynamics belief to better handle uncertainty, with theoretical and practical algorithms for offline RL.

Findings

Achieves state-of-the-art performance on benchmark tasks.

Demonstrates effective handling of dynamics uncertainty.

Provides theoretical guarantees for policy improvement.

Abstract

Model-based offline reinforcement learning (RL) aims to find highly rewarding policy, by leveraging a previously collected static dataset and a dynamics model. While the dynamics model learned through reuse of the static dataset, its generalization ability hopefully promotes policy learning if properly utilized. To that end, several works propose to quantify the uncertainty of predicted dynamics, and explicitly apply it to penalize reward. However, as the dynamics and the reward are intrinsically different factors in context of MDP, characterizing the impact of dynamics uncertainty through reward penalty may incur unexpected tradeoff between model utilization and risk avoidance. In this work, we instead maintain a belief distribution over dynamics, and evaluate/optimize policy through biased sampling from the belief. The sampling procedure, biased towards pessimism, is derived based on an alternating Markov game formulation of offline RL. We formally show that the biased sampling naturally induces an updated dynamics belief with policy-dependent reweighting factor, termed Pessimism-Modulated Dynamics Belief. To improve policy, we devise an iterative regularized policy optimization algorithm for the game, with guarantee of monotonous improvement under certain condition. To make practical, we further devise an offline RL algorithm to approximately find the solution. Empirical results show that the proposed approach achieves state-of-the-art performance on a wide range of benchmark tasks.