LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights

TL;DR

LD-Pruner is a novel structured pruning method for Latent Diffusion Models that reduces inference time and parameters while preserving performance, enabling deployment on resource-limited devices.

Contribution

The paper introduces LD-Pruner, a task-agnostic, latent space-based pruning technique specifically designed for LDMs, addressing their unique challenges and improving efficiency.

Findings

Reduced inference time of Stable Diffusion by 34.9%.

Improved FID score by 5.2% on MS-COCO T2I benchmark.

Effective pruning across text-to-image, image, and audio generation tasks.

Abstract

Latent Diffusion Models (LDMs) have emerged as powerful generative models, known for delivering remarkable results under constrained computational resources. However, deploying LDMs on resource-limited devices remains a complex issue, presenting challenges such as memory consumption and inference speed. To address this issue, we introduce LD-Pruner, a novel performance-preserving structured pruning method for compressing LDMs. Traditional pruning methods for deep neural networks are not tailored to the unique characteristics of LDMs, such as the high computational cost of training and the absence of a fast, straightforward and task-agnostic method for evaluating model performance. Our method tackles these challenges by leveraging the latent space during the pruning process, enabling us to effectively quantify the impact of pruning on model performance, independently of the task at hand. This targeted pruning of components with minimal impact on the output allows for faster convergence during training, as the model has less information to re-learn, thereby addressing the high computational cost of training. Consequently, our approach achieves a compressed model that offers improved inference speed and reduced parameter count, while maintaining minimal performance degradation. We demonstrate the effectiveness of our approach on three different tasks: text-to-image (T2I) generation, Unconditional Image Generation (UIG) and Unconditional Audio Generation (UAG). Notably, we reduce the inference time of Stable Diffusion (SD) by 34.9% while simultaneously improving its FID by 5.2% on MS-COCO T2I benchmark. This work paves the way for more efficient pruning methods for LDMs, enhancing their applicability.