Q-VDiT: Towards Accurate Quantization and Distillation of Video-Generation Diffusion Transformers

TL;DR

Q-VDiT is a specialized quantization framework for video diffusion transformers that improves efficiency and maintains high-quality video generation by addressing quantization errors and preserving spatiotemporal correlations.

Contribution

The paper introduces Q-VDiT, a novel quantization method with Token-aware Quantization Estimator and Temporal Maintenance Distillation tailored for video diffusion transformers.

Findings

Achieves a scene consistency score of 23.40, outperforming previous methods.

Reduces model size and computational complexity for edge deployment.

Sets a new benchmark in quantized video generation quality.

Abstract

Diffusion transformers (DiT) have demonstrated exceptional performance in video generation. However, their large number of parameters and high computational complexity limit their deployment on edge devices. Quantization can reduce storage requirements and accelerate inference by lowering the bit-width of model parameters. Yet, existing quantization methods for image generation models do not generalize well to video generation tasks. We identify two primary challenges: the loss of information during quantization and the misalignment between optimization objectives and the unique requirements of video generation. To address these challenges, we present Q-VDiT, a quantization framework specifically designed for video DiT models. From the quantization perspective, we propose the Token-aware Quantization Estimator (TQE), which compensates for quantization errors in both the token and feature dimensions. From the optimization perspective, we introduce Temporal Maintenance Distillation (TMD), which preserves the spatiotemporal correlations between frames and enables the optimization of each frame with respect to the overall video context. Our W3A6 Q-VDiT achieves a scene consistency of 23.40, setting a new benchmark and outperforming current state-of-the-art quantization methods by 1.9$\times$. Code will be available at https://github.com/cantbebetter2/Q-VDiT.