Provably Sample-Efficient RL with Side Information about Latent Dynamics

TL;DR

This paper introduces TASID, a transfer RL algorithm that leverages abstract side information about latent dynamics to learn robust policies efficiently, with sample complexity independent of state space size.

Contribution

The paper formalizes a transfer RL setting with side information about latent dynamics and proposes TASID, an algorithm with polynomial sample complexity independent of the number of states.

Findings

TASID outperforms algorithms requiring full simulators in synthetic experiments.

The algorithm achieves sample efficiency polynomial in the horizon.

It effectively utilizes abstract knowledge to handle environment stochasticity.

Abstract

We study reinforcement learning (RL) in settings where observations are high-dimensional, but where an RL agent has access to abstract knowledge about the structure of the state space, as is the case, for example, when a robot is tasked to go to a specific room in a building using observations from its own camera, while having access to the floor plan. We formalize this setting as transfer reinforcement learning from an abstract simulator, which we assume is deterministic (such as a simple model of moving around the floor plan), but which is only required to capture the target domain's latent-state dynamics approximately up to unknown (bounded) perturbations (to account for environment stochasticity). Crucially, we assume no prior knowledge about the structure of observations in the target domain except that they can be used to identify the latent states (but the decoding map is unknown). Under these assumptions, we present an algorithm, called TASID, that learns a robust policy in the target domain, with sample complexity that is polynomial in the horizon, and independent of the number of states, which is not possible without access to some prior knowledge. In synthetic experiments, we verify various properties of our algorithm and show that it empirically outperforms transfer RL algorithms that require access to "full simulators" (i.e., those that also simulate observations).