HadaNorm: Diffusion Transformer Quantization through Mean-Centered Transformations

TL;DR

HadaNorm introduces a linear transformation combining normalization and Hadamard transforms to improve quantization of diffusion transformers, reducing errors and enabling efficient deployment on resource-limited devices.

Contribution

HadaNorm is a novel linear transformation that enhances post-training quantization of diffusion transformers by mitigating outliers through normalization and Hadamard transforms.

Findings

HadaNorm outperforms state-of-the-art quantization methods.

It effectively reduces quantization error in transformer components.

Enables aggressive activation quantization for diffusion models.

Abstract

Diffusion models represent the cutting edge in image generation, but their high memory and computational demands hinder deployment on resource-constrained devices. Post-Training Quantization (PTQ) offers a promising solution by reducing the bitwidth of matrix operations. However, standard PTQ methods struggle with outliers, and achieving higher compression often requires transforming model weights and activations before quantization. In this work, we propose HadaNorm, a novel linear transformation that extends existing approaches by both normalizing channels activations and applying Hadamard transforms to effectively mitigate outliers and enable aggressive activation quantization. We demonstrate that HadaNorm consistently reduces quantization error across the various components of transformer blocks, outperforming state-of-the-art methods.