ConvRot: Rotation-Based Plug-and-Play 4-bit Quantization for Diffusion Transformers

TL;DR

ConvRot introduces a novel rotation-based quantization method that significantly reduces memory and computation for diffusion transformers, enabling efficient 4-bit inference without retraining while maintaining high image quality.

Contribution

The paper presents ConvRot, a group-wise rotation-based quantization technique using Hadamard transform, and ConvLinear4bit, a plug-and-play module for efficient W4A4 inference in diffusion transformers.

Findings

Achieves 2.26× speedup and 4.05× memory reduction on FLUX.1-dev.

Maintains high image fidelity with 4-bit quantization.

First application of rotation-based quantization for plug-and-play diffusion transformer inference.

Abstract

Diffusion transformers have demonstrated strong capabilities in generating high-quality images. However, as model size increases, the growing memory footprint and inference latency pose significant challenges for practical deployment. Recent studies in large language models (LLMs) show that rotation-based techniques can smooth outliers and enable 4-bit quantization, but these approaches often incur substantial overhead and struggle with row-wise outliers in diffusion transformers. To address these challenges, we propose ConvRot, a group-wise rotation-based quantization method that leverages regular Hadamard transform (RHT) to suppress both row-wise and column-wise outliers while reducing complexity from quadratic to linear. Building on this, we design ConvLinear4bit, a plug-and-play module that integrates rotation, quantization, GEMM, and dequantization, enabling W4A4 inference without retraining and preserving visual quality. Experiments on FLUX.1-dev demonstrate a 2.26$\times$ speedup and 4.05$\times$ memory reduction while maintaining image fidelity. To our knowledge, this is the first application of rotation-based quantization for plug-and-play W4A4 inference in diffusion transformers.