Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL

TL;DR

This paper introduces Behavior-Adaptive Q-Learning (BAQ), a framework that facilitates a stable and efficient transition from offline to online reinforcement learning by leveraging implicit behavioral models and adaptive constraints.

Contribution

BAQ is a novel framework that uses implicit behavioral models and dual-objective loss to improve offline-to-online RL transition stability and performance.

Findings

BAQ outperforms prior methods on standard benchmarks.

BAQ achieves faster recovery and higher robustness.

BAQ stabilizes early online updates and accelerates adaptation.

Abstract

Offline reinforcement learning (RL) enables training from fixed data without online interaction, but policies learned offline often struggle when deployed in dynamic environments due to distributional shift and unreliable value estimates on unseen state-action pairs. We introduce Behavior-Adaptive Q-Learning (BAQ), a framework designed to enable a smooth and reliable transition from offline to online RL. The key idea is to leverage an implicit behavioral model derived from offline data to provide a behavior-consistency signal during online fine-tuning. BAQ incorporates a dual-objective loss that (i) aligns the online policy toward the offline behavior when uncertainty is high, and (ii) gradually relaxes this constraint as more confident online experience is accumulated. This adaptive mechanism reduces error propagation from out-of-distribution estimates, stabilizes early online updates, and accelerates adaptation to new scenarios. Across standard benchmarks, BAQ consistently outperforms prior offline-to-online RL approaches, achieving faster recovery, improved robustness, and higher overall performance. Our results demonstrate that implicit behavior adaptation is a principled and practical solution for reliable real-world policy deployment.