Tackling Over-pruning in Variational Autoencoders

TL;DR

This paper addresses the issue of factor over-pruning in variational autoencoders by proposing the epitomic VAE, which groups latent factors to improve utilization and generalization, demonstrated on MNIST and TFD datasets.

Contribution

The paper introduces the epitomic variational autoencoder (eVAE), a novel model that groups latent factors to prevent inactive units and enhance model capacity.

Findings

eVAE outperforms traditional VAE in utilization of latent space

eVAE demonstrates better generalization on benchmark datasets

Qualitative and quantitative improvements shown on MNIST and TFD

Abstract

Variational autoencoders (VAE) are directed generative models that learn factorial latent variables. As noted by Burda et al. (2015), these models exhibit the problem of factor over-pruning where a significant number of stochastic factors fail to learn anything and become inactive. This can limit their modeling power and their ability to learn diverse and meaningful latent representations. In this paper, we evaluate several methods to address this problem and propose a more effective model-based approach called the epitomic variational autoencoder (eVAE). The so-called epitomes of this model are groups of mutually exclusive latent factors that compete to explain the data. This approach helps prevent inactive units since each group is pressured to explain the data. We compare the approaches with qualitative and quantitative results on MNIST and TFD datasets. Our results show that eVAE makes efficient use of model capacity and generalizes better than VAE.