When and how can inexact generative models still sample from the data manifold?

TL;DR

This paper investigates why certain dynamical generative models produce samples that stay on the data manifold despite learning errors, revealing the role of Lyapunov vector alignment in support robustness.

Contribution

It provides a dynamical systems analysis explaining the robustness of the data support in generative models and introduces a practical condition for achieving this robustness.

Findings

Infinitesimal learning errors affect the density only on the data manifold.

Alignment of Lyapunov vectors with the data boundary ensures support robustness.

The alignment condition can be efficiently computed and improves tangent space estimation.

Abstract

A curious phenomenon observed in some dynamical generative models is the following: despite learning errors in the score function or the drift vector field, the generated samples appear to shift \emph{along} the support of the data distribution but not \emph{away} from it. In this work, we investigate this phenomenon of \emph{robustness of the support} by taking a dynamical systems approach on the generating stochastic/deterministic process. Our perturbation analysis of the probability flow reveals that infinitesimal learning errors cause the predicted density to be different from the target density only on the data manifold for a wide class of generative models. Further, what is the dynamical mechanism that leads to the robustness of the support? We show that the alignment of the top Lyapunov vectors (most sensitive infinitesimal perturbation directions) with the tangent spaces along the boundary of the data manifold leads to robustness and prove a sufficient condition on the dynamics of the generating process to achieve this alignment. Moreover, the alignment condition is efficient to compute and, in practice, for robust generative models, automatically leads to accurate estimates of the tangent bundle of the data manifold. Using a finite-time linear perturbation analysis on samples paths as well as probability flows, our work complements and extends existing works on obtaining theoretical guarantees for generative models from a stochastic analysis, statistical learning and uncertainty quantification points of view. Our results apply across different dynamical generative models, such as conditional flow-matching and score-based generative models, and for different target distributions that may or may not satisfy the manifold hypothesis.