LIFT and PLACE: A Simple, Stable, and Effective Knowledge Distillation Framework for Lightweight Diffusion Models

TL;DR

This paper introduces LIFT and PLACE, a novel coarse-to-fine knowledge distillation framework that significantly improves the training stability and performance of lightweight diffusion models, even under extreme compression.

Contribution

The authors propose LIFT and PLACE, innovative methods for stable and effective knowledge distillation in diffusion models, addressing complex denoising processes and spatially non-uniform errors.

Findings

LIFT and PLACE outperform conventional KD across various models and tasks.

The framework maintains stability and achieves low FID scores with extremely compressed students.

Effective in both image/latent spaces and flow-based models.

Abstract

We demonstrate that in knowledge distillation for diffusion models, the teacher network's highly complex denoising process - stemming from its substantially larger capacity - poses a significant challenge for the student model to faithfully mimic. To address this problem, we propose a coarse-to-fine distillation framework with LInear FiTtingbased distillation (LIFT) and Piecewise Local Adaptive Coefficient Estimation (PLACE). First, LIFT decomposes the objective into a "coarse" alignment and a "fine" refinement. The student is then trained on coarse alignment before proceeding to hard refinement. Second, PLACE extends LIFT to address spatially non-uniform errors by partitioning outputs into error-based groups, providing locally adaptive guidance. Our experiments show that LIFT and PLACE is effective across diffusion spaces (image/latent), backbones (U-Net/DiT), tasks (unconditional/conditional), datasets, and even extends to flow-based models such as MMDiT (SD3). Furthermore, under extreme compression with a 1.3M-parameter student (only 1.6% of the teacher), conventional KD fails to provide sufficient guidance for stable training, with FID scores often degrading to 50-200+, but our method remains stably convergent and achieves an FID of 15.73.