Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL

TL;DR

This paper introduces a novel state-adaptive regularization technique for offline RL that dynamically adjusts regularization strength based on state quality, improving policy learning by balancing extrapolation error mitigation and policy flexibility.

Contribution

It proposes a selective, state-adaptive regularization method that enhances existing offline RL algorithms by tailoring regularization to state-specific data quality, leading to better performance.

Findings

Outperforms state-of-the-art methods on D4RL benchmarks

Effectively balances regularization to reduce extrapolation errors

Improves offline-to-online policy transfer

Abstract

Offline reinforcement learning (RL) aims to learn an effective policy from a static dataset. To alleviate extrapolation errors, existing studies often uniformly regularize the value function or policy updates across all states. However, due to substantial variations in data quality, the fixed regularization strength often leads to a dilemma: Weak regularization strength fails to address extrapolation errors and value overestimation, while strong regularization strength shifts policy learning toward behavior cloning, impeding potential performance enabled by Bellman updates. To address this issue, we propose the selective state-adaptive regularization method for offline RL. Specifically, we introduce state-adaptive regularization coefficients to trust state-level Bellman-driven results, while selectively applying regularization on high-quality actions, aiming to avoid performance degradation caused by tight constraints on low-quality actions. By establishing a connection between the representative value regularization method, CQL, and explicit policy constraint methods, we effectively extend selective state-adaptive regularization to these two mainstream offline RL approaches. Extensive experiments demonstrate that the proposed method significantly outperforms the state-of-the-art approaches in both offline and offline-to-online settings on the D4RL benchmark.