Structure by Architecture: Structured Representations without Regularization

TL;DR

This paper introduces a novel autoencoder architecture that learns structured, independent latent representations without regularization, improving generative and downstream task performance.

Contribution

The authors propose a new autoencoder design that captures hierarchical, independent latent variables without regularization, avoiding common trade-offs in generative modeling.

Findings

Improved generative quality and disentanglement.

Effective hierarchical latent representations learned.

Enhanced extrapolation capabilities.

Abstract

We study the problem of self-supervised structured representation learning using autoencoders for downstream tasks such as generative modeling. Unlike most methods which rely on matching an arbitrary, relatively unstructured, prior distribution for sampling, we propose a sampling technique that relies solely on the independence of latent variables, thereby avoiding the trade-off between reconstruction quality and generative performance typically observed in VAEs. We design a novel autoencoder architecture capable of learning a structured representation without the need for aggressive regularization. Our structural decoders learn a hierarchy of latent variables, thereby ordering the information without any additional regularization or supervision. We demonstrate how these models learn a representation that improves results in a variety of downstream tasks including generation, disentanglement, and extrapolation using several challenging and natural image datasets.