From Structure to Detail: Hierarchical Distillation for Efficient Diffusion Model

TL;DR

This paper introduces Hierarchical Distillation, a novel framework combining trajectory and distribution-based methods to significantly improve the efficiency and quality of diffusion models, enabling high-fidelity single-step generation.

Contribution

The paper proposes a synergistic Hierarchical Distillation framework that leverages trajectory distillation for structural initialization and introduces Adaptive Weighted Discriminator for enhanced detail refinement.

Findings

Achieves state-of-the-art FID of 2.26 on ImageNet 256x256 with a single step.

Demonstrates high-quality results on high-resolution text-to-image benchmarks.

Establishes a new paradigm for efficient, high-fidelity diffusion models.

Abstract

The inference latency of diffusion models remains a critical barrier to their real-time application. While trajectory-based and distribution-based step distillation methods offer solutions, they present a fundamental trade-off. Trajectory-based methods preserve global structure but act as a "lossy compressor", sacrificing high-frequency details. Conversely, distribution-based methods can achieve higher fidelity but often suffer from mode collapse and unstable training. This paper recasts them from independent paradigms into synergistic components within our novel Hierarchical Distillation (HD) framework. We leverage trajectory distillation not as a final generator, but to establish a structural ``sketch", providing a near-optimal initialization for the subsequent distribution-based refinement stage. This strategy yields an ideal initial distribution that enhances the ceiling of overall performance. To further improve quality, we introduce and refine the adversarial training process. We find standard discriminator structures are ineffective at refining an already high-quality generator. To overcome this, we introduce the Adaptive Weighted Discriminator (AWD), tailored for the HD pipeline. By dynamically allocating token weights, AWD focuses on local imperfections, enabling efficient detail refinement. Our approach demonstrates state-of-the-art performance across diverse tasks. On ImageNet $256\times256$, our single-step model achieves an FID of 2.26, rivaling its 250-step teacher. It also achieves promising results on the high-resolution text-to-image MJHQ benchmark, proving its generalizability. Our method establishes a robust new paradigm for high-fidelity, single-step diffusion models.