Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learning

TL;DR

This paper introduces TCE, a framework for cross-domain offline reinforcement learning that uses a dual score-based generative model to synthesize target-aligned transitions, improving policy adaptation across different environments.

Contribution

The paper proposes a novel Target-aligned Coverage Expansion framework that effectively leverages source data and generates target-consistent transitions to reduce distributional mismatch.

Findings

TCE outperforms existing cross-domain offline RL methods in various environments.

The dual score-based generative model effectively synthesizes target-aligned transitions.

Theoretical analysis guides the decision-making process for data utilization.

Abstract

Cross-domain offline reinforcement learning aims to adapt a policy from a source domain to a target domain using only pre-collected datasets, where environment dynamics may differ. A key challenge is to leverage source data while reducing distributional mismatch, particularly when the target dataset is extremely limited. To address this, we propose Target-aligned Coverage Expansion (TCE), a framework that decides how source data should be used, either by directly incorporating target-near transitions or by expanding state coverage through target-aligned generation, guided by theoretical analysis. TCE builds on a dual score-based generative model to synthesize target-consistent transitions over an expanded state region. Extensive experiments across diverse cross-domain environments show that TCE consistently outperforms state-of-the-art cross-domain offline RL baselines.