AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer

TL;DR

AdaLog introduces an adaptive logarithmic quantizer for Vision Transformers that optimizes activation quantization, significantly improving efficiency and accuracy across multiple vision tasks with hardware-friendly implementation.

Contribution

The paper proposes AdaLog, a novel adaptive logarithm quantizer with a fast search strategy, tailored for post-training quantization of ViT activations, addressing distribution challenges and hardware constraints.

Findings

Effective quantization of post-Softmax and post-GELU activations.

Improved accuracy and efficiency on various ViT architectures.

Versatile performance across classification, detection, and segmentation tasks.

Abstract

Vision Transformer (ViT) has become one of the most prevailing fundamental backbone networks in the computer vision community. Despite the high accuracy, deploying it in real applications raises critical challenges including the high computational cost and inference latency. Recently, the post-training quantization (PTQ) technique has emerged as a promising way to enhance ViT's efficiency. Nevertheless, existing PTQ approaches for ViT suffer from the inflexible quantization on the post-Softmax and post-GELU activations that obey the power-law-like distributions. To address these issues, we propose a novel non-uniform quantizer, dubbed the Adaptive Logarithm AdaLog (AdaLog) quantizer. It optimizes the logarithmic base to accommodate the power-law-like distribution of activations, while simultaneously allowing for hardware-friendly quantization and de-quantization. By employing the bias reparameterization, the AdaLog quantizer is applicable to both the post-Softmax and post-GELU activations. Moreover, we develop an efficient Fast Progressive Combining Search (FPCS) strategy to determine the optimal logarithm base for AdaLog, as well as the scaling factors and zero points for the uniform quantizers. Extensive experimental results on public benchmarks demonstrate the effectiveness of our approach for various ViT-based architectures and vision tasks including classification, object detection, and instance segmentation. Code is available at https://github.com/GoatWu/AdaLog.