Sig-DEG for Distillation: Making Diffusion Models Faster and Lighter

TL;DR

This paper introduces Sig-DEG, a signature-based generator that distills diffusion models to significantly reduce inference steps while maintaining high-quality generative results.

Contribution

Sig-DEG is a novel approach that uses partial signatures to efficiently approximate diffusion processes, enabling faster and lighter diffusion model inference.

Findings

Reduces inference steps by an order of magnitude

Maintains competitive generation quality

Efficient approximation of stochastic differential equations

Abstract

Diffusion models have achieved state-of-the-art results in generative modelling but remain computationally intensive at inference time, often requiring thousands of discretization steps. To this end, we propose Sig-DEG (Signature-based Differential Equation Generator), a novel generator for distilling pre-trained diffusion models, which can universally approximate the backward diffusion process at a coarse temporal resolution. Inspired by high-order approximations of stochastic differential equations (SDEs), Sig-DEG leverages partial signatures to efficiently summarize Brownian motion over sub-intervals and adopts a recurrent structure to enable accurate global approximation of the SDE solution. Distillation is formulated as a supervised learning task, where Sig-DEG is trained to match the outputs of a fine-resolution diffusion model on a coarse time grid. During inference, Sig-DEG enables fast generation, as the partial signature terms can be simulated exactly without requiring fine-grained Brownian paths. Experiments demonstrate that Sig-DEG achieves competitive generation quality while reducing the number of inference steps by an order of magnitude. Our results highlight the effectiveness of signature-based approximations for efficient generative modeling.