Multi-Step Consistency Models: Fast Generation with Theoretical Guarantees

TL;DR

This paper provides a theoretical foundation for consistency models, demonstrating they can generate high-quality samples rapidly with provable guarantees, even in non-smooth data settings, surpassing existing methods.

Contribution

It offers the first theoretical analysis of multi-step consistency models, establishing convergence rates and showing their efficiency and accuracy in both smooth and non-smooth data distributions.

Findings

Achieves KL divergence of O(ε^2) with O(log(d/ε)) steps.

In non-smooth data, convergence with O(d log(d/ε)) steps.

Best in class convergence rates for non-smooth cases.

Abstract

Consistency models have recently emerged as a compelling alternative to traditional SDE-based diffusion models. They offer a significant acceleration in generation by producing high-quality samples in very few steps. Despite their empirical success, a proper theoretic justification for their speed-up is still lacking. In this work, we address the gap by providing a theoretical analysis of consistency models capable of mapping inputs at a given time to arbitrary points along the reverse trajectory. We show that one can achieve a KL divergence of order $ O(\varepsilon^2) $ using only $ O\left(\log\left(\frac{d}{\varepsilon}\right)\right) $ iterations with a constant step size. Additionally, under minimal assumptions on the data distribution (non smooth case) an increasingly common setting in recent diffusion model analyses we show that a similar KL convergence guarantee can be obtained, with the number of steps scaling as $ O\left(d \log\left(\frac{d}{\varepsilon}\right)\right) $. Going further, we also provide a theoretical analysis for estimation of such consistency models, concluding that accurate learning is feasible using small discretization steps, both in smooth and non-smooth settings. Notably, our results for the non-smooth case yield best in class convergence rates compared to existing SDE or ODE based analyses under minimal assumptions.