Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling

TL;DR

This paper introduces a generative behavior modeling approach for offline reinforcement learning that improves action diversity and performance by decoupling policy learning from explicit parameterization, leveraging advanced generative models.

Contribution

It proposes a novel decoupled generative modeling framework for offline RL, enhancing diversity and effectiveness over traditional policy models.

Findings

Achieves competitive or superior performance on D4RL benchmarks.

Successfully learns from heterogeneous datasets with multiple strategies.

Outperforms previous methods in complex tasks like AntMaze.

Abstract

In offline reinforcement learning, weighted regression is a common method to ensure the learned policy stays close to the behavior policy and to prevent selecting out-of-sample actions. In this work, we show that due to the limited distributional expressivity of policy models, previous methods might still select unseen actions during training, which deviates from their initial motivation. To address this problem, we adopt a generative approach by decoupling the learned policy into two parts: an expressive generative behavior model and an action evaluation model. The key insight is that such decoupling avoids learning an explicitly parameterized policy model with a closed-form expression. Directly learning the behavior policy allows us to leverage existing advances in generative modeling, such as diffusion-based methods, to model diverse behaviors. As for action evaluation, we combine our method with an in-sample planning technique to further avoid selecting out-of-sample actions and increase computational efficiency. Experimental results on D4RL datasets show that our proposed method achieves competitive or superior performance compared with state-of-the-art offline RL methods, especially in complex tasks such as AntMaze. We also empirically demonstrate that our method can successfully learn from a heterogeneous dataset containing multiple distinctive but similarly successful strategies, whereas previous unimodal policies fail.