HQ-DM: Single Hadamard Transformation-Based Quantization-Aware Training for Low-Bit Diffusion Models

TL;DR

This paper introduces HQ-DM, a quantization-aware training method using Single Hadamard Transformation to effectively reduce activation outliers, enabling low-bit diffusion models with minimal performance loss for image generation tasks.

Contribution

The paper proposes a novel Single Hadamard Transformation-based quantization framework that improves low-bit diffusion model performance by mitigating activation outliers during inference.

Findings

W4A4 scheme improves Inception Score by 12.8%.

W4A3 scheme improves Inception Score by 467.73%.

Supports INT convolution operations without amplifying weight outliers.

Abstract

Diffusion models have demonstrated significant applications in the field of image generation. However, their high computational and memory costs pose challenges for deployment. Model quantization has emerged as a promising solution to reduce storage overhead and accelerate inference. Nevertheless, existing quantization methods for diffusion models struggle to mitigate outliers in activation matrices during inference, leading to substantial performance degradation under low-bit quantization scenarios. To address this, we propose HQ-DM, a novel Quantization-Aware Training framework that applies Single Hadamard Transformation to activation matrices. This approach effectively reduces activation outliers while preserving model performance under quantization. Compared to traditional Double Hadamard Transformation, our proposed scheme offers distinct advantages by seamlessly supporting INT convolution operations while preventing the amplification of weight outliers. For conditional generation on the ImageNet 256x256 dataset using the LDM-4 model, our W4A4 and W4A3 quantization schemes improve the Inception Score by 12.8% and 467.73%, respectively, over the existing state-of-the-art method.