Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples

TL;DR

This paper introduces a theoretical framework for offline reinforcement learning that optimally balances limited target data with auxiliary source data, providing performance guarantees and empirical validation on benchmark tasks.

Contribution

It presents the first theoretical analysis of dataset weighting in offline RL, establishing performance bounds and optimal weight computation methods.

Findings

Performance bounds depend on source data quality and target sample size.

Optimal dataset weights can be computed in closed form.

Empirical results validate theoretical guarantees on benchmarks.

Abstract

Offline reinforcement learning (RL) learns effective policies from a static target dataset. The performance of state-of-the-art offline RL algorithms notwithstanding, it relies on the size of the target dataset, and it degrades if limited samples in the target dataset are available, which is often the case in real-world applications. To address this issue, domain adaptation that leverages auxiliary samples from related source datasets (such as simulators) can be beneficial. However, establishing the optimal way to trade off the limited target dataset and the large-but-biased source dataset while ensuring provably theoretical guarantees remains an open challenge. To the best of our knowledge, this paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the performance of offline RL. In particular, we establish performance bounds and the existence of the optimal weight, which can be computed in closed form under simplifying assumptions. We also provide algorithmic guarantees in terms of convergence to a neighborhood of the optimum. Notably, these results depend on the quality of the source dataset and the number of samples in the target dataset. Our empirical results on the well-known Procgen and MuJoCo benchmarks substantiate the theoretical contributions in this work.