No Representation without Transformation

TL;DR

This paper extends variational autoencoders to explicitly model transformations in the latent space, enabling interpretable higher order objects and improved out-of-distribution classification performance.

Contribution

It introduces a hierarchical graphical model that jointly infers higher order objects and their transformations, enhancing interpretability and classification accuracy.

Findings

Latent transformations reflect interpretable properties in observation space.

The model achieves comparable generative performance to standard VAEs without transformations.

Outperforms baselines significantly on out-of-distribution classification tasks.

Abstract

We extend the framework of variational autoencoders to represent transformations explicitly in the latent space. In the family of hierarchical graphical models that emerges, the latent space is populated by higher order objects that are inferred jointly with the latent representations they act on. To explicitly demonstrate the effect of these higher order objects, we show that the inferred latent transformations reflect interpretable properties in the observation space. Furthermore, the model is structured in such a way that in the absence of transformations, we can run inference and obtain generative capabilities comparable with standard variational autoencoders. Finally, utilizing the trained encoder, we outperform the baselines by a wide margin on a challenging out-of-distribution classification task.