Glance: Accelerating Diffusion Models with 1 Sample

TL;DR

This paper introduces Glance, a phase-aware acceleration method for diffusion models that uses lightweight adapters trained with minimal data, achieving significant speedups with strong generalization.

Contribution

It proposes a novel phase-aware acceleration strategy combined with lightweight LoRA adapters trained on only one sample, reducing retraining costs and maintaining quality.

Findings

Achieves up to 5x acceleration over base models.

Adapters trained with only 1 sample on a single V100.

Maintains high visual quality across benchmarks.

Abstract

Diffusion models have achieved remarkable success in image generation, yet their deployment remains constrained by the heavy computational cost and the need for numerous inference steps. Previous efforts on fewer-step distillation attempt to skip redundant steps by training compact student models, yet they often suffer from heavy retraining costs and degraded generalization. In this work, we take a different perspective: we accelerate smartly, not evenly, applying smaller speedups to early semantic stages and larger ones to later redundant phases. We instantiate this phase-aware strategy with two experts that specialize in slow and fast denoising phases. Surprisingly, instead of investing massive effort in retraining student models, we find that simply equipping the base model with lightweight LoRA adapters achieves both efficient acceleration and strong generalization. We refer to these two adapters as Slow-LoRA and Fast-LoRA. Through extensive experiments, our method achieves up to 5 acceleration over the base model while maintaining comparable visual quality across diverse benchmarks. Remarkably, the LoRA experts are trained with only 1 samples on a single V100 within one hour, yet the resulting models generalize strongly on unseen prompts.