ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers

TL;DR

This paper introduces ADFQ-ViT, a novel post-training quantization framework for Vision Transformers that effectively handles activation distribution challenges, significantly improving low-bit quantization accuracy across multiple vision tasks.

Contribution

The paper proposes new quantization methods tailored for ViT activation distributions, including Per-Patch Outlier-aware Quantizer and Shift-Log2 Quantizer, with module-wise optimization for enhanced accuracy.

Findings

Achieves 10.23% higher Top-1 accuracy at 4-bit quantization of ViT-B on ImageNet.

Significant improvements in image classification, object detection, and segmentation tasks.

Outperforms various baseline quantization methods in accuracy at low-bit settings.

Abstract

Vision Transformers (ViTs) have exhibited exceptional performance across diverse computer vision tasks, while their substantial parameter size incurs significantly increased memory and computational demands, impeding effective inference on resource-constrained devices. Quantization has emerged as a promising solution to mitigate these challenges, yet existing methods still suffer from significant accuracy loss at low-bit. We attribute this issue to the distinctive distributions of post-LayerNorm and post-GELU activations within ViTs, rendering conventional hardware-friendly quantizers ineffective, particularly in low-bit scenarios. To address this issue, we propose a novel framework called Activation-Distribution-Friendly post-training Quantization for Vision Transformers, ADFQ-ViT. Concretely, we introduce the Per-Patch Outlier-aware Quantizer to tackle irregular outliers in post-LayerNorm activations. This quantizer refines the granularity of the uniform quantizer to a per-patch level while retaining a minimal subset of values exceeding a threshold at full-precision. To handle the non-uniform distributions of post-GELU activations between positive and negative regions, we design the Shift-Log2 Quantizer, which shifts all elements to the positive region and then applies log2 quantization. Moreover, we present the Attention-score enhanced Module-wise Optimization which adjusts the parameters of each quantizer by reconstructing errors to further mitigate quantization error. Extensive experiments demonstrate ADFQ-ViT provides significant improvements over various baselines in image classification, object detection, and instance segmentation tasks at 4-bit. Specifically, when quantizing the ViT-B model to 4-bit, we achieve a 10.23% improvement in Top-1 accuracy on the ImageNet dataset.