Neighbor Embedding Variational Autoencoder

TL;DR

This paper introduces NE-VAE, a novel variational autoencoder that explicitly constrains the encoder to maintain meaningful latent representations, effectively preventing posterior collapse and enhancing latent space quality.

Contribution

The paper proposes NE-VAE, a new model that enforces local input relationships in the latent space, improving latent organization and reducing posterior collapse without complex modifications.

Findings

NE-VAE produces more active and meaningful latent dimensions.

It significantly reduces posterior collapse compared to existing VAEs.

The model is easily integrated into existing frameworks without added complexity.

Abstract

Being one of the most popular generative framework, variational autoencoders(VAE) are known to suffer from a phenomenon termed posterior collapse, i.e. the latent variational distributions collapse to the prior, especially when a strong decoder network is used. In this work, we analyze the latent representation of collapsed VAEs, and proposed a novel model, neighbor embedding VAE(NE-VAE), which explicitly constraints the encoder to encode inputs close in the input space to be close in the latent space. We observed that for VAE variants that report similar ELBO, KL divergence or even mutual information scores may still behave quite differently in the latent organization. In our experiments, NE-VAE can produce qualitatively different latent representations with majority of the latent dimensions remained active, which may benefit downstream latent space optimization tasks. NE-VAE can prevent posterior collapse to a much greater extent than it's predecessors, and can be easily plugged into any autoencoder framework, without introducing addition model components and complex training routines.