Offline Reinforcement Learning with Adaptive Behavior Regularization

TL;DR

This paper introduces adaptive behavior regularization (ABR), a novel method for offline reinforcement learning that dynamically balances policy cloning and improvement, leading to improved performance on standard benchmarks.

Contribution

The paper proposes ABR, a simple sample-based regularization technique that adaptively balances the trade-off between policy cloning and improvement in offline RL.

Findings

ABR achieves competitive or superior results on D4RL benchmarks.

ABR effectively balances policy cloning and enhancement.

The method outperforms existing offline RL algorithms in various tasks.

Abstract

Offline reinforcement learning (RL) defines a sample-efficient learning paradigm, where a policy is learned from static and previously collected datasets without additional interaction with the environment. The major obstacle to offline RL is the estimation error arising from evaluating the value of out-of-distribution actions. To tackle this problem, most existing offline RL methods attempt to acquire a policy both ``close" to the behaviors contained in the dataset and sufficiently improved over them, which requires a trade-off between two possibly conflicting targets. In this paper, we propose a novel approach, which we refer to as adaptive behavior regularization (ABR), to balance this critical trade-off. By simply utilizing a sample-based regularization, ABR enables the policy to adaptively adjust its optimization objective between cloning and improving over the policy used to generate the dataset. In the evaluation on D4RL datasets, a widely adopted benchmark for offline reinforcement learning, ABR can achieve improved or competitive performance compared to existing state-of-the-art algorithms.