Learning to Integrate Diffusion ODEs by Averaging the Derivatives

TL;DR

This paper introduces secant losses for learning to integrate diffusion ODEs, improving inference speed and stability by leveraging derivative-integral relationships, achieving state-of-the-art results with fewer steps.

Contribution

It proposes secant losses based on geometric principles to enhance diffusion ODE integration, enabling faster inference with stable training and superior performance.

Findings

Achieves 2.14 FID with 10 steps on CIFAR-10

Attains 1.96 FID with 8 steps on ImageNet-256x256

Secant approach improves speed and stability of diffusion models

Abstract

To accelerate diffusion model inference, numerical solvers perform poorly at extremely small steps, while distillation techniques often introduce complexity and instability. This work presents an intermediate strategy, balancing performance and cost, by learning ODE integration using loss functions derived from the derivative-integral relationship, inspired by Monte Carlo integration and Picard iteration. From a geometric perspective, the losses operate by gradually extending the tangent to the secant, thus are named as secant losses. The target of secant losses is the same as that of diffusion models, or the diffusion model itself, leading to great training stability. By fine-tuning or distillation, the secant version of EDM achieves a $10$-step FID of $2.14$ on CIFAR-10, while the secant version of SiT-XL/2 attains a $4$-step FID of $2.27$ and an $8$-step FID of $1.96$ on ImageNet-$256\times256$. Code is available at https://github.com/poppuppy/secant-expectation.