Offline Reinforcement Learning with On-Policy Q-Function Regularization

TL;DR

This paper introduces a novel offline RL method that regularizes towards the behavior policy's Q-function, improving stability and performance by leveraging more reliable Q-function estimates to mitigate extrapolation error.

Contribution

It proposes regularizing towards the behavior policy's Q-function instead of the policy itself, using SARSA-style estimates for better reliability and handling of extrapolation errors.

Findings

Strong performance on D4RL benchmarks

Effective mitigation of extrapolation error

Improved stability over existing methods

Abstract

The core challenge of offline reinforcement learning (RL) is dealing with the (potentially catastrophic) extrapolation error induced by the distribution shift between the history dataset and the desired policy. A large portion of prior work tackles this challenge by implicitly/explicitly regularizing the learning policy towards the behavior policy, which is hard to estimate reliably in practice. In this work, we propose to regularize towards the Q-function of the behavior policy instead of the behavior policy itself, under the premise that the Q-function can be estimated more reliably and easily by a SARSA-style estimate and handles the extrapolation error more straightforwardly. We propose two algorithms taking advantage of the estimated Q-function through regularizations, and demonstrate they exhibit strong performance on the D4RL benchmarks.