Entropic Auto-Encoding via Implicit Free-Energy Minimization

TL;DR

This paper introduces Entropic Autoencoders (EAEs), a novel framework that mitigates posterior collapse in variational autoencoders by implicitly modeling the prior through free energy minimization, leading to more informative and diverse latent representations.

Contribution

The paper proposes EAEs, which use entropy and ensemble methods to implicitly define the prior, avoiding explicit prior imposition and improving latent space quality.

Findings

EAEs learn non-Gaussian, multimodal latent distributions.

EAEs capture complex data structures like reaction-diffusion dynamics.

EAEs identify implicit categorical distinctions and hierarchical features.

Abstract

Despite their ubiquity, variational autoencoders (VAEs) inherently suffer from posterior collapse, a failure mode in which latent variables are effectively ignored. This failure arises because explicit prior imposition drives optimization toward loss landscape regions corresponding to uninformative latent representations. Here, we introduce Entropic Autoencoders (EAEs), a framework in which reconstruction loss is the only explicit objective, and entropy generates the latent variables' prior implicitly through a free energy-minimizing ensemble of encoders. This ensemble biases learning toward high-volume regions of near-optimal solutions, while decoder updates direct the search trajectories toward informative latent representations. We demonstrate that EAEs mitigate posterior collapse by learning non-Gaussian, multimodal latent distributions that yield diverse, data-consistent generations and preserve different forms of underlying structure in the data. As a proof-of-concept, we show that an EAE captures a superposition of the known low-dimensional dynamics of a reaction-diffusion process. Then, we show that an EAE identifies implicit categorical distinctions in MNIST latent representations, and displays a hierarchical understanding of facial structure on the CelebA dataset, from an "all-human" face to individual-dependent features.