DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization

TL;DR

This paper introduces DMQ, a novel post-training quantization method for diffusion models that effectively reduces quantization errors and improves image generation quality at low bit-widths by addressing outliers and critical denoising steps.

Contribution

DMQ combines Learned Equivalent Scaling and channel-wise Power-of-Two Scaling with adaptive timestep weighting and a voting algorithm to enhance low-bit quantization of diffusion models.

Findings

Outperforms existing PTQ methods at low bit-widths

Maintains high image quality and model stability

Effectively reduces quantization errors in diffusion models

Abstract

Diffusion models have achieved remarkable success in image generation but come with significant computational costs, posing challenges for deployment in resource-constrained environments. Recent post-training quantization (PTQ) methods have attempted to mitigate this issue by focusing on the iterative nature of diffusion models. However, these approaches often overlook outliers, leading to degraded performance at low bit-widths. In this paper, we propose a DMQ which combines Learned Equivalent Scaling (LES) and channel-wise Power-of-Two Scaling (PTS) to effectively address these challenges. Learned Equivalent Scaling optimizes channel-wise scaling factors to redistribute quantization difficulty between weights and activations, reducing overall quantization error. Recognizing that early denoising steps, despite having small quantization errors, crucially impact the final output due to error accumulation, we incorporate an adaptive timestep weighting scheme to prioritize these critical steps during learning. Furthermore, identifying that layers such as skip connections exhibit high inter-channel variance, we introduce channel-wise Power-of-Two Scaling for activations. To ensure robust selection of PTS factors even with small calibration set, we introduce a voting algorithm that enhances reliability. Extensive experiments demonstrate that our method significantly outperforms existing works, especially at low bit-widths such as W4A6 (4-bit weight, 6-bit activation) and W4A8, maintaining high image generation quality and model stability. The code is available at https://github.com/LeeDongYeun/dmq.