Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning

TL;DR

This paper introduces A2PR, a novel offline reinforcement learning method that adaptively guides policy regularization using advantage estimates and VAE-generated actions, improving performance on suboptimal datasets.

Contribution

A2PR is the first method to adaptively select high-advantage actions for policy regularization, balancing conservatism and policy improvement in offline RL.

Findings

Achieves state-of-the-art results on D4RL benchmarks.

Effectively mitigates value overestimation issues.

Performs well on suboptimal mixed datasets.

Abstract

In offline reinforcement learning, the challenge of out-of-distribution (OOD) is pronounced. To address this, existing methods often constrain the learned policy through policy regularization. However, these methods often suffer from the issue of unnecessary conservativeness, hampering policy improvement. This occurs due to the indiscriminate use of all actions from the behavior policy that generates the offline dataset as constraints. The problem becomes particularly noticeable when the quality of the dataset is suboptimal. Thus, we propose Adaptive Advantage-guided Policy Regularization (A2PR), obtaining high-advantage actions from an augmented behavior policy combined with VAE to guide the learned policy. A2PR can select high-advantage actions that differ from those present in the dataset, while still effectively maintaining conservatism from OOD actions. This is achieved by harnessing the VAE capacity to generate samples matching the distribution of the data points. We theoretically prove that the improvement of the behavior policy is guaranteed. Besides, it effectively mitigates value overestimation with a bounded performance gap. Empirically, we conduct a series of experiments on the D4RL benchmark, where A2PR demonstrates state-of-the-art performance. Furthermore, experimental results on additional suboptimal mixed datasets reveal that A2PR exhibits superior performance. Code is available at https://github.com/ltlhuuu/A2PR.