ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation

TL;DR

ViDiT-Q introduces a specialized post-training quantization method for diffusion transformers, significantly reducing memory and computational costs while maintaining high visual quality in image and video generation tasks.

Contribution

The paper presents ViDiT-Q, a novel quantization scheme specifically designed for diffusion transformers, enabling efficient deployment with minimal quality loss.

Findings

Achieves W8A8 and W4A8 quantization with negligible quality degradation.

Provides 2-2.5x memory savings and 1.4-1.7x latency speedup.

Effectively applies to various text-to-image and video diffusion models.

Abstract

Diffusion transformers have demonstrated remarkable performance in visual generation tasks, such as generating realistic images or videos based on textual instructions. However, larger model sizes and multi-frame processing for video generation lead to increased computational and memory costs, posing challenges for practical deployment on edge devices. Post-Training Quantization (PTQ) is an effective method for reducing memory costs and computational complexity. When quantizing diffusion transformers, we find that existing quantization methods face challenges when applied to text-to-image and video tasks. To address these challenges, we begin by systematically analyzing the source of quantization error and conclude with the unique challenges posed by DiT quantization. Accordingly, we design an improved quantization scheme: ViDiT-Q (Video & Image Diffusion Transformer Quantization), tailored specifically for DiT models. We validate the effectiveness of ViDiT-Q across a variety of text-to-image and video models, achieving W8A8 and W4A8 with negligible degradation in visual quality and metrics. Additionally, we implement efficient GPU kernels to achieve practical 2-2.5x memory saving and a 1.4-1.7x end-to-end latency speedup.