An Analysis on Quantizing Diffusion Transformers

TL;DR

This paper introduces a novel, optimization-free post-training quantization method for diffusion transformers, significantly reducing model size and inference cost without sacrificing performance, demonstrated through image generation tasks.

Contribution

It pioneers an efficient PTQ approach for transformer-only diffusion models, employing single-step activation calibration and group-wise weight quantization without optimization.

Findings

Effective low-bit quantization achieved without optimization

Reduced inference memory and computational requirements

Maintained image generation quality in experiments

Abstract

Diffusion Models (DMs) utilize an iterative denoising process to transform random noise into synthetic data. Initally proposed with a UNet structure, DMs excel at producing images that are virtually indistinguishable with or without conditioned text prompts. Later transformer-only structure is composed with DMs to achieve better performance. Though Latent Diffusion Models (LDMs) reduce the computational requirement by denoising in a latent space, it is extremely expensive to inference images for any operating devices due to the shear volume of parameters and feature sizes. Post Training Quantization (PTQ) offers an immediate remedy for a smaller storage size and more memory-efficient computation during inferencing. Prior works address PTQ of DMs on UNet structures have addressed the challenges in calibrating parameters for both activations and weights via moderate optimization. In this work, we pioneer an efficient PTQ on transformer-only structure without any optimization. By analysing challenges in quantizing activations and weights for diffusion transformers, we propose a single-step sampling calibration on activations and adapt group-wise quantization on weights for low-bit quantization. We demonstrate the efficiency and effectiveness of proposed methods with preliminary experiments on conditional image generation.