PFDiff: Training-Free Acceleration of Diffusion Models Combining Past and Future Scores

TL;DR

PFDiff introduces a training-free, timestep-skipping strategy that enhances the efficiency of diffusion model sampling by reducing function evaluations and correcting discretization errors, applicable across various pre-trained models.

Contribution

It proposes a novel training-free method combining past and future scores with momentum-inspired updates to accelerate diffusion sampling.

Findings

Achieves 16.46 FID with 4 NFE on ImageNet 64x64

Surpasses previous training-free acceleration methods

Effective across multiple pre-trained diffusion models

Abstract

Diffusion Probabilistic Models (DPMs) have shown remarkable potential in image generation, but their sampling efficiency is hindered by the need for numerous denoising steps. Most existing solutions accelerate the sampling process by proposing fast ODE solvers. However, the inevitable discretization errors of the ODE solvers are significantly magnified when the number of function evaluations (NFE) is fewer. In this work, we propose PFDiff, a novel training-free and orthogonal timestep-skipping strategy, which enables existing fast ODE solvers to operate with fewer NFE. Specifically, PFDiff initially utilizes score replacement from past time steps to predict a ``springboard". Subsequently, it employs this ``springboard" along with foresight updates inspired by Nesterov momentum to rapidly update current intermediate states. This approach effectively reduces unnecessary NFE while correcting for discretization errors inherent in first-order ODE solvers. Experimental results demonstrate that PFDiff exhibits flexible applicability across various pre-trained DPMs, particularly excelling in conditional DPMs and surpassing previous state-of-the-art training-free methods. For instance, using DDIM as a baseline, we achieved 16.46 FID (4 NFE) compared to 138.81 FID with DDIM on ImageNet 64x64 with classifier guidance, and 13.06 FID (10 NFE) on Stable Diffusion with 7.5 guidance scale. Code is available at \url{https://github.com/onefly123/PFDiff}.