Cross-Domain Offline Policy Adaptation via Selective Transition Correction

TL;DR

This paper introduces the Selective Transition Correction (STC) algorithm for cross-domain offline reinforcement learning, effectively aligning source data with target dynamics to improve policy adaptation.

Contribution

The paper proposes a novel method that modifies source domain data using inverse policy and reward models, combined with a forward dynamics model, to better match target domain dynamics.

Findings

STC outperforms existing baselines in environments with dynamics shifts.

The approach effectively aligns source data with target dynamics, improving policy learning.

Experimental results demonstrate the robustness of STC across various environments.

Abstract

It remains a critical challenge to adapt policies across domains with mismatched dynamics in reinforcement learning (RL). In this paper, we study cross-domain offline RL, where an offline dataset from another similar source domain can be accessed to enhance policy learning upon a target domain dataset. Directly merging the two datasets may lead to suboptimal performance due to potential dynamics mismatches. Existing approaches typically mitigate this issue through source domain transition filtering or reward modification, which, however, may lead to insufficient exploitation of the valuable source domain data. Instead, we propose to modify the source domain data into the target domain data. To that end, we leverage an inverse policy model and a reward model to correct the actions and rewards of source transitions, explicitly achieving alignment with the target dynamics. Since limited data may result in inaccurate model training, we further employ a forward dynamics model to retain corrected samples that better match the target dynamics than the original transitions. Consequently, we propose the Selective Transition Correction (STC) algorithm, which enables reliable usage of source domain data for policy adaptation. Experiments on various environments with dynamics shifts demonstrate that STC achieves superior performance against existing baselines.