Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shifts

TL;DR

This paper introduces DROCO, a novel offline RL algorithm that enhances robustness against dynamics shifts during both training and testing, addressing a key gap in cross-domain offline RL.

Contribution

The paper proposes a dual-robust framework with a new Bellman operator and techniques to improve test-time robustness in cross-domain offline RL.

Findings

DROCO outperforms baseline methods in various dynamics shift scenarios.

The RCB operator improves test-time robustness without sacrificing train-time stability.

Techniques like dynamic value penalty and Huber loss mitigate value estimation errors.

Abstract

Single-domain offline reinforcement learning (RL) often suffers from limited data coverage, while cross-domain offline RL handles this issue by leveraging additional data from other domains with dynamics shifts. However, existing studies primarily focus on train-time robustness (handling dynamics shifts from training data), neglecting the test-time robustness against dynamics perturbations when deployed in practical scenarios. In this paper, we investigate dual (both train-time and test-time) robustness against dynamics shifts in cross-domain offline RL. We first empirically show that the policy trained with cross-domain offline RL exhibits fragility under dynamics perturbations during evaluation, particularly when target domain data is limited. To address this, we introduce a novel robust cross-domain Bellman (RCB) operator, which enhances test-time robustness against dynamics perturbations while staying conservative to the out-of-distribution dynamics transitions, thus guaranteeing the train-time robustness. To further counteract potential value overestimation or underestimation caused by the RCB operator, we introduce two techniques, the dynamic value penalty and the Huber loss, into our framework, resulting in the practical \textbf{D}ual-\textbf{RO}bust \textbf{C}ross-domain \textbf{O}ffline RL (DROCO) algorithm. Extensive empirical results across various dynamics shift scenarios show that DROCO outperforms strong baselines and exhibits enhanced robustness to dynamics perturbations.