BK-SDM: A Lightweight, Fast, and Cheap Version of Stable Diffusion

TL;DR

This paper introduces BK-SDM, a lightweight and efficient version of Stable Diffusion that uses block pruning and feature distillation to significantly reduce model size and inference time while maintaining competitive image generation quality.

Contribution

The authors propose a novel architectural reduction method for SDMs using block pruning and distillation, enabling low-cost training and deployment on edge devices.

Findings

Achieved 30-50% reduction in model size, MACs, and latency.

Compact models can imitate original SDMs with limited resources.

Enabled deployment on edge devices with 4-second inference.

Abstract

Text-to-image (T2I) generation with Stable Diffusion models (SDMs) involves high computing demands due to billion-scale parameters. To enhance efficiency, recent studies have reduced sampling steps and applied network quantization while retaining the original architectures. The lack of architectural reduction attempts may stem from worries over expensive retraining for such massive models. In this work, we uncover the surprising potential of block pruning and feature distillation for low-cost general-purpose T2I. By removing several residual and attention blocks from the U-Net of SDMs, we achieve 30%~50% reduction in model size, MACs, and latency. We show that distillation retraining is effective even under limited resources: using only 13 A100 days and a tiny dataset, our compact models can imitate the original SDMs (v1.4 and v2.1-base with over 6,000 A100 days). Benefiting from the transferred knowledge, our BK-SDMs deliver competitive results on zero-shot MS-COCO against larger multi-billion parameter models. We further demonstrate the applicability of our lightweight backbones in personalized generation and image-to-image translation. Deployment of our models on edge devices attains 4-second inference. Code and models can be found at: https://github.com/Nota-NetsPresso/BK-SDM