Towards More Accurate Diffusion Model Acceleration with A Timestep Tuner

TL;DR

This paper introduces a timestep tuner that improves the accuracy and speed of diffusion model sampling by optimizing the integral direction at each step, significantly enhancing performance especially with fewer denoising steps.

Contribution

The proposed timestep tuner adaptively finds more accurate integral directions for diffusion sampling, boosting existing acceleration methods without significant additional cost.

Findings

Improves FID from 9.65 to 6.07 on LSUN Bedroom with 10 steps

Enhances various state-of-the-art acceleration methods

Efficient plug-in design that requires minimal training

Abstract

A diffusion model, which is formulated to produce an image using thousands of denoising steps, usually suffers from a slow inference speed. Existing acceleration algorithms simplify the sampling by skipping most steps yet exhibit considerable performance degradation. By viewing the generation of diffusion models as a discretized integral process, we argue that the quality drop is partly caused by applying an inaccurate integral direction to a timestep interval. To rectify this issue, we propose a \textbf{timestep tuner} that helps find a more accurate integral direction for a particular interval at the minimum cost. Specifically, at each denoising step, we replace the original parameterization by conditioning the network on a new timestep, enforcing the sampling distribution towards the real one. Extensive experiments show that our plug-in design can be trained efficiently and boost the inference performance of various state-of-the-art acceleration methods, especially when there are few denoising steps. For example, when using 10 denoising steps on LSUN Bedroom dataset, we improve the FID of DDIM from 9.65 to 6.07, simply by adopting our method for a more appropriate set of timesteps. Code is available at \href{https://github.com/THU-LYJ-Lab/time-tuner}{https://github.com/THU-LYJ-Lab/time-tuner}.