BiDM: Pushing the Limit of Quantization for Diffusion Models

TL;DR

This paper introduces BiDM, a novel method for fully binarizing diffusion models to significantly reduce storage and computation costs while maintaining high-quality image generation, pushing quantization to the 1-bit limit.

Contribution

BiDM is the first approach to fully binarize diffusion models, employing Timestep-friendly Binary Structure and Space Patched Distillation to preserve performance at extreme quantization.

Findings

Achieves an FID of 22.74 on LSUN-Bedrooms 256×256, outperforming previous binarization methods.

Provides up to 28.0 times storage savings and 52.7 times OPs reduction.

Demonstrates effective full binarization of diffusion models with minimal performance degradation.

Abstract

Diffusion models (DMs) have been significantly developed and widely used in various applications due to their excellent generative qualities. However, the expensive computation and massive parameters of DMs hinder their practical use in resource-constrained scenarios. As one of the effective compression approaches, quantization allows DMs to achieve storage saving and inference acceleration by reducing bit-width while maintaining generation performance. However, as the most extreme quantization form, 1-bit binarization causes the generation performance of DMs to face severe degradation or even collapse. This paper proposes a novel method, namely BiDM, for fully binarizing weights and activations of DMs, pushing quantization to the 1-bit limit. From a temporal perspective, we introduce the Timestep-friendly Binary Structure (TBS), which uses learnable activation binarizers and cross-timestep feature connections to address the highly timestep-correlated activation features of DMs. From a spatial perspective, we propose Space Patched Distillation (SPD) to address the difficulty of matching binary features during distillation, focusing on the spatial locality of image generation tasks and noise estimation networks. As the first work to fully binarize DMs, the W1A1 BiDM on the LDM-4 model for LSUN-Bedrooms 256$\times$256 achieves a remarkable FID of 22.74, significantly outperforming the current state-of-the-art general binarization methods with an FID of 59.44 and invalid generative samples, and achieves up to excellent 28.0 times storage and 52.7 times OPs savings. The code is available at https://github.com/Xingyu-Zheng/BiDM .