Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning

TL;DR

This paper introduces an adaptive behavior cloning regularization method that improves the stability and efficiency of offline-to-online reinforcement learning, achieving state-of-the-art results on the D4RL benchmark.

Contribution

It proposes an adaptive weighting scheme for behavior cloning loss and uses a randomized ensemble of Q functions to enhance online fine-tuning.

Findings

Achieves state-of-the-art offline-to-online RL performance on D4RL.

Adaptive behavior cloning improves training stability.

Ensemble of Q functions increases sample efficiency.

Abstract

Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. However, depending on the quality of the offline dataset, such pre-trained agents may have limited performance and would further need to be fine-tuned online by interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. While constraints enforced by offline RL methods such as a behaviour cloning loss prevent this to an extent, these constraints also significantly slow down online fine-tuning by forcing the agent to stay close to the behavior policy. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Moreover, we use a randomized ensemble of Q functions to further increase the sample efficiency of online fine-tuning by performing a large number of learning updates. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark. Code is available: \url{https://github.com/zhaoyi11/adaptive_bc}.