Deep Transfer $Q$-Learning for Offline Non-Stationary Reinforcement Learning

TL;DR

This paper introduces a novel transfer deep Q-learning method for non-stationary reinforcement learning, leveraging neural networks and a re-weighted sampling strategy to improve decision-making across dynamic, diverse populations.

Contribution

It develops a new transfer learning framework for non-stationary RL with neural networks, including a re-weighted sampling procedure and theoretical guarantees, addressing limitations of naive sample pooling.

Findings

Outperforms naive pooling in non-stationary RL scenarios

Theoretically guarantees transferability with neural networks

Validated on synthetic and real datasets

Abstract

In dynamic decision-making scenarios across business and healthcare, leveraging sample trajectories from diverse populations can significantly enhance reinforcement learning (RL) performance for specific target populations, especially when sample sizes are limited. While existing transfer learning methods primarily focus on linear regression settings, they lack direct applicability to reinforcement learning algorithms. This paper pioneers the study of transfer learning for dynamic decision scenarios modeled by non-stationary finite-horizon Markov decision processes, utilizing neural networks as powerful function approximators and backward inductive learning. We demonstrate that naive sample pooling strategies, effective in regression settings, fail in Markov decision processes.To address this challenge, we introduce a novel ``re-weighted targeting procedure'' to construct ``transferable RL samples'' and propose ``transfer deep $Q^*$-learning'', enabling neural network approximation with theoretical guarantees. We assume that the reward functions are transferable and deal with both situations in which the transition densities are transferable or nontransferable. Our analytical techniques for transfer learning in neural network approximation and transition density transfers have broader implications, extending to supervised transfer learning with neural networks and domain shift scenarios. Empirical experiments on both synthetic and real datasets corroborate the advantages of our method, showcasing its potential for improving decision-making through strategically constructing transferable RL samples in non-stationary reinforcement learning contexts.