IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers

TL;DR

IPTQ-ViT introduces a fully integer-only post-training quantization framework for vision transformers, utilizing novel approximation functions and a unified metric to improve accuracy without retraining, suitable for resource-limited environments.

Contribution

The paper proposes a new PTQ method for vision transformers that achieves fully integer-only inference without retraining, using optimized approximation functions and a unified selection metric.

Findings

Achieves up to 6.44% accuracy improvement over previous PTQ methods.

Outperforms partial floating-point PTQ methods under W8A8 and W4A8.

Comparable accuracy and latency to integer-only QAT methods.

Abstract

Previous Quantization-Aware Training (QAT) methods for vision transformers rely on expensive retraining to recover accuracy loss in non-linear layer quantization, limiting their use in resource-constrained environments. In contrast, existing Post-Training Quantization (PTQ) methods either partially quantize non-linear functions or adjust activation distributions to maintain accuracy but fail to achieve fully integer-only inference. In this paper, we introduce IPTQ-ViT, a novel PTQ framework for fully integer-only vision transformers without retraining. We present approximation functions: a polynomial-based GELU optimized for vision data and a bit-shifting-based Softmax designed to improve approximation accuracy in PTQ. In addition, we propose a unified metric integrating quantization sensitivity, perturbation, and computational cost to select the optimal approximation function per activation layer. IPTQ-ViT outperforms previous PTQ methods, achieving up to 6.44\%p (avg. 1.78\%p) top-1 accuracy improvement for image classification, 1.0 mAP for object detection. IPTQ-ViT outperforms partial floating-point PTQ methods under W8A8 and W4A8, and achieves accuracy and latency comparable to integer-only QAT methods. We plan to release our code https://github.com/gihwan-kim/IPTQ-ViT.git.