Pluggable Pruning with Contiguous Layer Distillation for Diffusion Transformers

TL;DR

This paper introduces PPCL, a structured pruning framework for Diffusion Transformers that reduces parameters by 50% with minimal performance loss, enabling efficient deployment in resource-limited settings.

Contribution

The paper proposes a novel pluggable pruning method with a contiguous layer distillation scheme tailored for Diffusion Transformers, allowing flexible compression without retraining for each ratio.

Findings

Achieves 50% parameter reduction with less than 3% performance degradation.

Maintains high-quality image generation after pruning.

Enables resource-efficient deployment of Diffusion Transformers.

Abstract

Diffusion Transformers (DiTs) have shown exceptional performance in image generation, yet their large parameter counts incur high computational costs, impeding deployment in resource-constrained settings. To address this, we propose Pluggable Pruning with Contiguous Layer Distillation (PPCL), a flexible structured pruning framework specifically designed for DiT architectures. First, we identify redundant layer intervals through a linear probing mechanism combined with the first-order differential trend analysis of similarity metrics. Subsequently, we propose a plug-and-play teacher-student alternating distillation scheme tailored to integrate depth-wise and width-wise pruning within a single training phase. This distillation framework enables flexible knowledge transfer across diverse pruning ratios, eliminating the need for per-configuration retraining. Extensive experiments on multiple Multi-Modal Diffusion Transformer architecture models demonstrate that PPCL achieves a 50\% reduction in parameter count compared to the full model, with less than 3\% degradation in key objective metrics. Notably, our method maintains high-quality image generation capabilities while achieving higher compression ratios, rendering it well-suited for resource-constrained environments. The open-source code, checkpoints for PPCL can be found at the following link: https://github.com/OPPO-Mente-Lab/Qwen-Image-Pruning.