Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning

TL;DR

This paper introduces LoDADA, a localized domain adaptation method for offline RL that clusters transitions to better handle local dynamics mismatch, improving policy learning across diverse environments.

Contribution

LoDADA is a novel clustering-based approach that estimates and filters source data based on localized dynamics discrepancies, enhancing data reuse in off-dynamics offline RL.

Findings

LoDADA outperforms existing methods in diverse environments.

It effectively leverages localized dynamics mismatch.

The method is scalable and computationally efficient.

Abstract

Off-dynamics offline reinforcement learning (RL) aims to learn a policy for a target domain using limited target data and abundant source data collected under different transition dynamics. Existing methods typically address dynamics mismatch either globally over the state space or via pointwise data filtering; these approaches can miss localized cross-domain similarities or incur high computational cost. We propose Localized Dynamics-Aware Domain Adaptation (LoDADA), which exploits localized dynamics mismatch to better reuse source data. LoDADA clusters transitions from source and target datasets and estimates cluster-level dynamics discrepancy via domain discrimination. Source transitions from clusters with small discrepancy are retained, while those from clusters with large discrepancy are filtered out. This yields a fine-grained and scalable data selection strategy that avoids overly coarse global assumptions and expensive per-sample filtering. We provide theoretical insights and extensive experiments across environments with diverse global and local dynamics shifts. Results show that LoDADA consistently outperforms state-of-the-art off-dynamics offline RL methods by better leveraging localized distribution mismatch.