Interpretable Representation Learning from Videos using Nonlinear Priors

TL;DR

This paper introduces a deep learning framework that incorporates nonlinear physical priors into video representation learning, enabling the generation of interpretable and physically plausible hypothetical videos.

Contribution

It extends the VAE prior to a nonlinear Additive Noise Model and develops a novel linearization method for stable training, allowing physical interpretability in video generation.

Findings

Successfully learned physical variables from real-world physics videos.

Generated plausible hypothetical videos by intervening on physical variables.

Validated on diverse physics scenarios like pendulums and pulsars.

Abstract

Learning interpretable representations of visual data is an important challenge, to make machines' decisions understandable to humans and to improve generalisation outside of the training distribution. To this end, we propose a deep learning framework where one can specify nonlinear priors for videos (e.g. of Newtonian physics) that allow the model to learn interpretable latent variables and use these to generate videos of hypothetical scenarios not observed at training time. We do this by extending the Variational Auto-Encoder (VAE) prior from a simple isotropic Gaussian to an arbitrary nonlinear temporal Additive Noise Model (ANM), which can describe a large number of processes (e.g. Newtonian physics). We propose a novel linearization method that constructs a Gaussian Mixture Model (GMM) approximating the prior, and derive a numerically stable Monte Carlo estimate of the KL divergence between the posterior and prior GMMs. We validate the method on different real-world physics videos including a pendulum, a mass on a spring, a falling object and a pulsar (rotating neutron star). We specify a physical prior for each experiment and show that the correct variables are learned. Once a model is trained, we intervene on it to change different physical variables (such as oscillation amplitude or adding air drag) to generate physically correct videos of hypothetical scenarios that were not observed previously.