DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing

TL;DR

DiTAS introduces a data-free post-training quantization method for diffusion transformers, employing activation smoothing and weight quantization techniques to enable efficient low-bit inference with minimal performance loss.

Contribution

The paper presents a novel quantization approach for diffusion transformers that combines activation smoothing, layer-wise optimization, and a training-free weight quantization module.

Findings

Enables 4-bit weight and 8-bit activation quantization for DiTs.

Maintains comparable performance to full-precision models after quantization.

Reduces implementation costs, facilitating deployment on resource-constrained devices.

Abstract

Diffusion Transformers (DiTs) have recently attracted significant interest from both industry and academia due to their enhanced capabilities in visual generation, surpassing the performance of traditional diffusion models that employ U-Net. However, the improved performance of DiTs comes at the expense of higher parameter counts and implementation costs, which significantly limits their deployment on resource-constrained devices like mobile phones. We propose DiTAS, a data-free post-training quantization (PTQ) method for efficient DiT inference. DiTAS relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations, leading to much lower quantization error under extremely low bitwidth. To further enhance the performance of the quantized DiT, we adopt the layer-wise grid search strategy to optimize the smoothing factor. Moreover, we integrate a training-free LoRA module for weight quantization, leveraging alternating optimization to minimize quantization errors without additional fine-tuning. Experimental results demonstrate that our approach enables 4-bit weight, 8-bit activation (W4A8) quantization for DiTs while maintaining comparable performance as the full-precision model.