Accelerating Score-based Generative Models with Preconditioned Diffusion Sampling

TL;DR

This paper introduces a preconditioned diffusion sampling method that significantly accelerates score-based generative models, reducing sampling time by up to 29 times without sacrificing quality, by addressing ill-conditioned curvature issues.

Contribution

The paper proposes a model-agnostic preconditioning approach for diffusion sampling, with theoretical convergence guarantees, to speed up score-based generative models without retraining.

Findings

PDS accelerates sampling by up to 29x on high-resolution images.

PDS maintains synthesis quality comparable to standard methods.

Theoretical proof ensures convergence to the original distribution.

Abstract

Score-based generative models (SGMs) have recently emerged as a promising class of generative models. However, a fundamental limitation is that their inference is very slow due to a need for many (e.g., 2000) iterations of sequential computations. An intuitive acceleration method is to reduce the sampling iterations which however causes severe performance degradation. We investigate this problem by viewing the diffusion sampling process as a Metropolis adjusted Langevin algorithm, which helps reveal the underlying cause to be ill-conditioned curvature. Under this insight, we propose a model-agnostic preconditioned diffusion sampling (PDS) method that leverages matrix preconditioning to alleviate the aforementioned problem. Crucially, PDS is proven theoretically to converge to the original target distribution of a SGM, no need for retraining. Extensive experiments on three image datasets with a variety of resolutions and diversity validate that PDS consistently accelerates off-the-shelf SGMs whilst maintaining the synthesis quality. In particular, PDS can accelerate by up to 29x on more challenging high resolution (1024x1024) image generation.