NewtonianVAE: Proportional Control and Goal Identification from Pixels via Physical Latent Spaces

TL;DR

This paper introduces NewtonianVAE, a latent dynamics model that facilitates proportional control from pixels, simplifying control tasks and enabling interpretable goal discovery in vision-based imitation learning.

Contribution

The paper presents a novel latent dynamics learning framework that induces proportional controlability in the latent space, improving control simplicity and interpretability.

Findings

Enables proportional control directly from pixel inputs.

Simplifies and accelerates behavioral cloning for vision-based controllers.

Provides interpretable goal discovery in imitation learning.

Abstract

Learning low-dimensional latent state space dynamics models has been a powerful paradigm for enabling vision-based planning and learning for control. We introduce a latent dynamics learning framework that is uniquely designed to induce proportional controlability in the latent space, thus enabling the use of much simpler controllers than prior work. We show that our learned dynamics model enables proportional control from pixels, dramatically simplifies and accelerates behavioural cloning of vision-based controllers, and provides interpretable goal discovery when applied to imitation learning of switching controllers from demonstration.