The Limits of Transfer Reinforcement Learning with Latent Low-rank Structure

TL;DR

This paper investigates the limits of transfer reinforcement learning using latent low-rank structures, proposing algorithms that leverage linear representations to reduce complexity and establishing their optimality bounds.

Contribution

It introduces a transfer-ability coefficient and algorithms for latent low-rank MDPs, achieving near-optimal regret bounds that depend on the transfer difficulty.

Findings

Algorithms effectively reduce dependence on state and action space sizes.

Proved minimax optimality of the algorithms in most settings.

Provided lower bounds matching the upper bounds for transfer learning.

Abstract

Many reinforcement learning (RL) algorithms are too costly to use in practice due to the large sizes $S, A$ of the problem's state and action space. To resolve this issue, we study transfer RL with latent low rank structure. We consider the problem of transferring a latent low rank representation when the source and target MDPs have transition kernels with Tucker rank $(S , d, A )$, $(S , S , d), (d, S, A )$, or $(d , d , d )$. In each setting, we introduce the transfer-ability coefficient $\alpha$ that measures the difficulty of representational transfer. Our algorithm learns latent representations in each source MDP and then exploits the linear structure to remove the dependence on $S, A $, or $S A$ in the target MDP regret bound. We complement our positive results with information theoretic lower bounds that show our algorithms (excluding the ($d, d, d$) setting) are minimax-optimal with respect to $\alpha$.