A solvable model of learning generative diffusion: theory and insights

TL;DR

This paper presents a theoretical analysis of learning flow-based generative models using auto-encoders, revealing how sample size influences the learned distribution and explaining phenomena like mode collapse.

Contribution

It provides a rigorous asymptotic characterization of the learned distribution in high-dimensional settings with low-dimensional structure, and analyzes mode collapse mechanisms.

Findings

Distribution convergence depends on training sample size

Mode collapse can occur during re-training on generated data

The model captures low-dimensional manifold structure

Abstract

In this manuscript, we consider the problem of learning a flow or diffusion-based generative model parametrized by a two-layer auto-encoder, trained with online stochastic gradient descent, on a high-dimensional target density with an underlying low-dimensional manifold structure. We derive a tight asymptotic characterization of low-dimensional projections of the distribution of samples generated by the learned model, ascertaining in particular its dependence on the number of training samples. Building on this analysis, we discuss how mode collapse can arise, and lead to model collapse when the generative model is re-trained on generated synthetic data.