On Incorporating Inductive Biases into VAEs

TL;DR

This paper introduces InteL-VAEs, a novel approach that uses intermediary latent variables to effectively incorporate inductive biases into variational autoencoders, improving both generative quality and learned representations.

Contribution

The paper proposes InteL-VAEs, a new method that directly enforces inductive biases via an intermediary latent space, overcoming limitations of prior prior-based approaches.

Findings

InteL-VAEs better enforce desired properties like sparsity and clustering.

They produce improved generative models and representations.

The method bypasses issues with non-Gaussian priors.

Abstract

We explain why directly changing the prior can be a surprisingly ineffective mechanism for incorporating inductive biases into VAEs, and introduce a simple and effective alternative approach: Intermediary Latent Space VAEs(InteL-VAEs). InteL-VAEs use an intermediary set of latent variables to control the stochasticity of the encoding process, before mapping these in turn to the latent representation using a parametric function that encapsulates our desired inductive bias(es). This allows us to impose properties like sparsity or clustering on learned representations, and incorporate human knowledge into the generative model. Whereas changing the prior only indirectly encourages behavior through regularizing the encoder, InteL-VAEs are able to directly enforce desired characteristics. Moreover, they bypass the computation and encoder design issues caused by non-Gaussian priors, while allowing for additional flexibility through training of the parametric mapping function. We show that these advantages, in turn, lead to both better generative models and better representations being learned.