Distilling Diffusion Models into Conditional GANs

TL;DR

This paper introduces a method to convert complex diffusion models into fast, single-step conditional GANs by using a novel distillation process and a perceptual loss in latent space, achieving high-quality image generation.

Contribution

The authors present a new diffusion distillation technique as image-to-image translation and introduce E-LatentLPIPS, a perceptual loss in latent space, enabling efficient and effective model compression.

Findings

Outperforms existing one-step diffusion distillation models on COCO benchmark

E-LatentLPIPS converges faster than other perceptual losses

The method maintains high image quality with significantly reduced inference time

Abstract

We propose a method to distill a complex multistep diffusion model into a single-step conditional GAN student model, dramatically accelerating inference, while preserving image quality. Our approach interprets diffusion distillation as a paired image-to-image translation task, using noise-to-image pairs of the diffusion model's ODE trajectory. For efficient regression loss computation, we propose E-LatentLPIPS, a perceptual loss operating directly in diffusion model's latent space, utilizing an ensemble of augmentations. Furthermore, we adapt a diffusion model to construct a multi-scale discriminator with a text alignment loss to build an effective conditional GAN-based formulation. E-LatentLPIPS converges more efficiently than many existing distillation methods, even accounting for dataset construction costs. We demonstrate that our one-step generator outperforms cutting-edge one-step diffusion distillation models -- DMD, SDXL-Turbo, and SDXL-Lightning -- on the zero-shot COCO benchmark.