Lipschitz Singularities in Diffusion Models

TL;DR

This paper investigates the Lipschitz properties of diffusion models, revealing singularities that affect stability, and proposes a new method, E-TSDM, to mitigate these issues, leading to improved performance and reduced FID scores.

Contribution

The paper provides theoretical and empirical analysis of Lipschitz singularities in diffusion models and introduces E-TSDM, a novel approach to alleviate these singularities near zero timesteps.

Findings

Lipschitz singularities are present near zero timesteps in diffusion models.

E-TSDM significantly reduces Lipschitz singularities and improves model performance.

Achieved over 33% reduction in FID scores for acceleration methods.

Abstract

Diffusion models, which employ stochastic differential equations to sample images through integrals, have emerged as a dominant class of generative models. However, the rationality of the diffusion process itself receives limited attention, leaving the question of whether the problem is well-posed and well-conditioned. In this paper, we explore a perplexing tendency of diffusion models: they often display the infinite Lipschitz property of the network with respect to time variable near the zero point. We provide theoretical proofs to illustrate the presence of infinite Lipschitz constants and empirical results to confirm it. The Lipschitz singularities pose a threat to the stability and accuracy during both the training and inference processes of diffusion models. Therefore, the mitigation of Lipschitz singularities holds great potential for enhancing the performance of diffusion models. To address this challenge, we propose a novel approach, dubbed E-TSDM, which alleviates the Lipschitz singularities of the diffusion model near the zero point of timesteps. Remarkably, our technique yields a substantial improvement in performance. Moreover, as a byproduct of our method, we achieve a dramatic reduction in the Fr\'echet Inception Distance of acceleration methods relying on network Lipschitz, including DDIM and DPM-Solver, by over 33%. Extensive experiments on diverse datasets validate our theory and method. Our work may advance the understanding of the general diffusion process, and also provide insights for the design of diffusion models.