Deep Generative Models through the Lens of the Manifold Hypothesis: A Survey and New Connections

TL;DR

This paper surveys deep generative models through the manifold hypothesis, explaining their successes and failures, and introduces new theoretical insights into their stability and representation capabilities.

Contribution

It provides the first survey of DGMs from the manifold perspective and introduces novel theoretical results on likelihood instability and Wasserstein interpretation of autoencoder-based models.

Findings

Likelihood instability is unavoidable in high dimensions for low-dimensional data.

Autoencoder representations in DGMs can be viewed as minimizing Wasserstein distance.

Manifold perspective clarifies why some DGMs outperform others.

Abstract

In recent years there has been increased interest in understanding the interplay between deep generative models (DGMs) and the manifold hypothesis. Research in this area focuses on understanding the reasons why commonly-used DGMs succeed or fail at learning distributions supported on unknown low-dimensional manifolds, as well as developing new models explicitly designed to account for manifold-supported data. This manifold lens provides both clarity as to why some DGMs (e.g. diffusion models and some generative adversarial networks) empirically surpass others (e.g. likelihood-based models such as variational autoencoders, normalizing flows, or energy-based models) at sample generation, and guidance for devising more performant DGMs. We carry out the first survey of DGMs viewed through this lens, making two novel contributions along the way. First, we formally establish that numerical instability of likelihoods in high ambient dimensions is unavoidable when modelling data with low intrinsic dimension. We then show that DGMs on learned representations of autoencoders can be interpreted as approximately minimizing Wasserstein distance: this result, which applies to latent diffusion models, helps justify their outstanding empirical results. The manifold lens provides a rich perspective from which to understand DGMs, and we aim to make this perspective more accessible and widespread.