Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

TL;DR

Embed to Control (E2C) is a deep generative model that learns locally linear latent dynamics from raw images, enabling effective control and long-term prediction of complex non-linear systems.

Contribution

The paper introduces E2C, a novel variational autoencoder-based model that constrains latent dynamics to be locally linear for improved control from raw images.

Findings

Supports long-term image sequence prediction

Demonstrates strong control performance on complex tasks

Learns effective latent representations for control

Abstract

We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.