MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization

TL;DR

MixDQ introduces a memory-efficient mixed-precision quantization framework for few-step text-to-image diffusion models, significantly reducing memory and latency while maintaining image quality and text alignment.

Contribution

The paper proposes a novel mixed-precision quantization method with specialized sensitivity analysis and bit-width allocation for diffusion models, enabling high compression without performance loss.

Findings

Achieves 3-4x reduction in model size and memory cost compared to FP16.

Maintains image quality and text alignment at W8A8 quantization.

Provides 1.45x speedup in inference latency.

Abstract

Diffusion models have achieved significant visual generation quality. However, their significant computational and memory costs pose challenge for their application on resource-constrained mobile devices or even desktop GPUs. Recent few-step diffusion models reduces the inference time by reducing the denoising steps. However, their memory consumptions are still excessive. The Post Training Quantization (PTQ) replaces high bit-width FP representation with low-bit integer values (INT4/8) , which is an effective and efficient technique to reduce the memory cost. However, when applying to few-step diffusion models, existing quantization methods face challenges in preserving both the image quality and text alignment. To address this issue, we propose an mixed-precision quantization framework - MixDQ. Firstly, We design specialized BOS-aware quantization method for highly sensitive text embedding quantization. Then, we conduct metric-decoupled sensitivity analysis to measure the sensitivity of each layer. Finally, we develop an integer-programming-based method to conduct bit-width allocation. While existing quantization methods fall short at W8A8, MixDQ could achieve W8A8 without performance loss, and W4A8 with negligible visual degradation. Compared with FP16, we achieve 3-4x reduction in model size and memory cost, and 1.45x latency speedup.