CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation

TL;DR

CoDi introduces a novel diffusion distillation method that enables high-fidelity, condition-aware image generation with significantly fewer steps, making real-time applications more feasible.

Contribution

The paper presents CoDi, a new approach that adapts pre-trained diffusion models for conditioned image generation with fewer steps, improving efficiency and quality.

Findings

Achieves state-of-the-art results with 1-4 steps across multiple tasks.

Effectively incorporates conditioning inputs without losing pre-trained knowledge.

Outperforms previous distillation methods in quality and speed.

Abstract

Large generative diffusion models have revolutionized text-to-image generation and offer immense potential for conditional generation tasks such as image enhancement, restoration, editing, and compositing. However, their widespread adoption is hindered by the high computational cost, which limits their real-time application. To address this challenge, we introduce a novel method dubbed CoDi, that adapts a pre-trained latent diffusion model to accept additional image conditioning inputs while significantly reducing the sampling steps required to achieve high-quality results. Our method can leverage architectures such as ControlNet to incorporate conditioning inputs without compromising the model's prior knowledge gained during large scale pre-training. Additionally, a conditional consistency loss enforces consistent predictions across diffusion steps, effectively compelling the model to generate high-quality images with conditions in a few steps. Our conditional-task learning and distillation approach outperforms previous distillation methods, achieving a new state-of-the-art in producing high-quality images with very few steps (e.g., 1-4) across multiple tasks, including super-resolution, text-guided image editing, and depth-to-image generation.