Deterministic training of generative autoencoders using invertible layers

TL;DR

This paper introduces AutoEncoders within Flows (AEF), a deterministic alternative to variational autoencoders that uses invertible layers for improved performance and sharper samples.

Contribution

It presents a new class of deterministic autoencoders with invertible architectures, offering higher likelihood and sample quality than VAEs, especially in low-dimensional latent spaces.

Findings

AEFs outperform VAEs in log-likelihood and sample sharpness

AEFs can be used as drop-in replacements for VAEs

AEFs produce substantially sharper samples

Abstract

In this work, we provide a deterministic alternative to the stochastic variational training of generative autoencoders. We refer to these new generative autoencoders as AutoEncoders within Flows (AEF), since the encoder and decoder are defined as affine layers of an overall invertible architecture. This results in a deterministic encoding of the data, as opposed to the stochastic encoding of VAEs. The paper introduces two related families of AEFs. The first family relies on a partition of the ambient space and is trained by exact maximum-likelihood. The second family exploits a deterministic expansion of the ambient space and is trained by maximizing the log-probability in this extended space. This latter case leaves complete freedom in the choice of encoder, decoder and prior architectures, making it a drop-in replacement for the training of existing VAEs and VAE-style models. We show that these AEFs can have strikingly higher performance than architecturally identical VAEs in terms of log-likelihood and sample quality, especially for low dimensional latent spaces. Importantly, we show that AEF samples are substantially sharper than VAE samples.