Extracting Latent State Representations with Linear Dynamics from Rich Observations

TL;DR

This paper presents an efficient method to identify linear latent dynamics within high-dimensional observations, such as images, and extends this to nonlinear mappings, aiding reinforcement learning in complex environments.

Contribution

It introduces an algorithm to extract linear latent subspaces from rich observations and extends it to nonlinear mappings, improving learning in high-dimensional RL tasks.

Findings

Successfully extracts linear latent dynamics from rich observations.

Empirically verifies the approach's effectiveness in simple settings.

Extends method to nonlinear mappings for broader applicability.

Abstract

Recently, many reinforcement learning techniques were shown to have provable guarantees in the simple case of linear dynamics, especially in problems like linear quadratic regulators. However, in practice, many reinforcement learning problems try to learn a policy directly from rich, high dimensional representations such as images. Even if there is an underlying dynamics that is linear in the correct latent representations (such as position and velocity), the rich representation is likely to be nonlinear and can contain irrelevant features. In this work we study a model where there is a hidden linear subspace in which the dynamics is linear. For such a model we give an efficient algorithm for extracting the linear subspace with linear dynamics. We then extend our idea to extracting a nonlinear mapping, and empirically verify the effectiveness of our approach in simple settings with rich observations.