Geometric Regularity in Deterministic Sampling Dynamics of Diffusion-based Generative Models

TL;DR

This paper uncovers a geometric regularity in the trajectories of diffusion-based generative models, showing they follow a low-dimensional, boomerang-shaped path, which can be exploited to improve sampling efficiency and image quality.

Contribution

The study reveals a universal geometric pattern in deterministic sampling trajectories of diffusion models and introduces a novel scheduling scheme leveraging this regularity for enhanced performance.

Findings

Trajectories lie in low-dimensional subspaces.

All trajectories share a boomerang shape.

The proposed scheduling improves image quality with fewer evaluations.

Abstract

Diffusion-based generative models employ stochastic differential equations (SDEs) and their equivalent probability flow ordinary differential equations (ODEs) to establish a smooth transformation between complex high-dimensional data distributions and tractable prior distributions. In this paper, we reveal a striking geometric regularity in the deterministic sampling dynamics of diffusion generative models: each simulated sampling trajectory along the gradient field lies within an extremely low-dimensional subspace, and all trajectories exhibit an almost identical boomerang shape, regardless of the model architecture, applied conditions, or generated content. We characterize several intriguing properties of these trajectories, particularly under closed-form solutions based on kernel-estimated data modeling. We also demonstrate a practical application of the discovered trajectory regularity by proposing a dynamic programming-based scheme to better align the sampling time schedule with the underlying trajectory structure. This simple strategy requires minimal modification to existing deterministic numerical solvers, incurs negligible computational overhead, and achieves superior image generation performance, especially in regions with only 5 - 10 function evaluations.