Noise Contrastive Variational Autoencoders

TL;DR

This paper investigates the posterior collapse problem in VAEs, introduces NC-VAE which uses noise contrastive estimation to prevent collapse, and demonstrates its effectiveness on image and text datasets.

Contribution

It proposes NC-VAE, a novel VAE variant that employs noise contrastive estimation to avoid posterior collapse, supported by theoretical proofs and empirical results.

Findings

NC-VAE cannot reach posterior collapse theoretically.

NC-VAE improves data reconstruction quality.

NC-VAE performs well on image and text datasets.

Abstract

We take steps towards understanding the "posterior collapse (PC)" difficulty in variational autoencoders (VAEs),~i.e. a degenerate optimum in which the latent codes become independent of their corresponding inputs. We rely on calculus of variations and theoretically explore a few popular VAE models, showing that PC always occurs for non-parametric encoders and decoders. Inspired by the popular noise contrastive estimation algorithm, we propose NC-VAE where the encoder discriminates between the latent codes of real data and of some artificially generated noise, in addition to encouraging good data reconstruction abilities. Theoretically, we prove that our model cannot reach PC and provide novel lower bounds. Our method is straightforward to implement and has the same run-time as vanilla VAE. Empirically, we showcase its benefits on popular image and text datasets.