Rich-Observation Reinforcement Learning with Continuous Latent Dynamics

TL;DR

This paper introduces RichCLD, a theoretical framework and efficient algorithm for reinforcement learning with high-dimensional observations governed by low-dimensional continuous latent dynamics, improving sample efficiency and reliability.

Contribution

The paper presents a novel framework and a new representation learning objective tailored for continuous latent dynamics in rich-observation RL, with theoretical guarantees and practical advantages.

Findings

The new algorithm is provably statistically and computationally efficient.

Empirical results favorably compare to prior representation learning schemes.

Insights into the statistical complexity of rich-observation RL with continuous dynamics.

Abstract

Sample-efficiency and reliability remain major bottlenecks toward wide adoption of reinforcement learning algorithms in continuous settings with high-dimensional perceptual inputs. Toward addressing these challenges, we introduce a new theoretical framework, RichCLD (Rich-Observation RL with Continuous Latent Dynamics), in which the agent performs control based on high-dimensional observations, but the environment is governed by low-dimensional latent states and Lipschitz continuous dynamics. Our main contribution is a new algorithm for this setting that is provably statistically and computationally efficient. The core of our algorithm is a new representation learning objective; we show that prior representation learning schemes tailored to discrete dynamics do not naturally extend to the continuous setting. Our new objective is amenable to practical implementation, and empirically, we find that it compares favorably to prior schemes in a standard evaluation protocol. We further provide several insights into the statistical complexity of the RichCLD framework, in particular proving that certain notions of Lipschitzness that admit sample-efficient learning in the absence of rich observations are insufficient in the rich-observation setting.