Qua$^2$SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Models

TL;DR

This paper introduces Qua$^2$SeDiMo, a framework for analyzing and optimizing quantization sensitivity in diffusion models, enabling high-quality mixed-precision quantization across various architectures.

Contribution

It provides explainable insights into quantization effects on different model layers, guiding effective mixed-precision quantization for diffusion models including U-Nets and Transformers.

Findings

Achieved 3.4-3.9 bit weight quantization on various diffusion models.

Outperformed existing methods in quantitative metrics and image quality.

Provided a systematic approach for quantization sensitivity analysis.

Abstract

Diffusion Models (DM) have democratized AI image generation through an iterative denoising process. Quantization is a major technique to alleviate the inference cost and reduce the size of DM denoiser networks. However, as denoisers evolve from variants of convolutional U-Nets toward newer Transformer architectures, it is of growing importance to understand the quantization sensitivity of different weight layers, operations and architecture types to performance. In this work, we address this challenge with Qua$^2$SeDiMo, a mixed-precision Post-Training Quantization framework that generates explainable insights on the cost-effectiveness of various model weight quantization methods for different denoiser operation types and block structures. We leverage these insights to make high-quality mixed-precision quantization decisions for a myriad of diffusion models ranging from foundational U-Nets to state-of-the-art Transformers. As a result, Qua$^2$SeDiMo can construct 3.4-bit, 3.9-bit, 3.65-bit and 3.7-bit weight quantization on PixArt-${\alpha}$, PixArt-${\Sigma}$, Hunyuan-DiT and SDXL, respectively. We further pair our weight-quantization configurations with 6-bit activation quantization and outperform existing approaches in terms of quantitative metrics and generative image quality.