Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers

TL;DR

Q-DiT introduces a novel post-training quantization method for diffusion transformers, significantly reducing computational costs while maintaining high image and video generation quality, through adaptive quantization techniques.

Contribution

The paper presents Q-DiT, a new PTQ framework that effectively handles variance in diffusion transformers using automatic granularity allocation and sample-wise dynamic activation quantization.

Findings

Achieves a 1.09 FID reduction on ImageNet W6A8 quantization.

Maintains high fidelity in W4A8 quantization for image and video generation.

Establishes new benchmarks for efficient quantization of diffusion transformers.

Abstract

Recent advancements in diffusion models, particularly the architectural transformation from UNet-based models to Diffusion Transformers (DiTs), significantly improve the quality and scalability of image and video generation. However, despite their impressive capabilities, the substantial computational costs of these large-scale models pose significant challenges for real-world deployment. Post-Training Quantization (PTQ) emerges as a promising solution, enabling model compression and accelerated inference for pretrained models, without the costly retraining. However, research on DiT quantization remains sparse, and existing PTQ frameworks, primarily designed for traditional diffusion models, tend to suffer from biased quantization, leading to notable performance degradation. In this work, we identify that DiTs typically exhibit significant spatial variance in both weights and activations, along with temporal variance in activations. To address these issues, we propose Q-DiT, a novel approach that seamlessly integrates two key techniques: automatic quantization granularity allocation to handle the significant variance of weights and activations across input channels, and sample-wise dynamic activation quantization to adaptively capture activation changes across both timesteps and samples. Extensive experiments conducted on ImageNet and VBench demonstrate the effectiveness of the proposed Q-DiT. Specifically, when quantizing DiT-XL/2 to W6A8 on ImageNet ($256 \times 256$), Q-DiT achieves a remarkable reduction in FID by 1.09 compared to the baseline. Under the more challenging W4A8 setting, it maintains high fidelity in image and video generation, establishing a new benchmark for efficient, high-quality quantization in DiTs. Code is available at \href{https://github.com/Juanerx/Q-DiT}{https://github.com/Juanerx/Q-DiT}.