Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction

TL;DR

This paper introduces ERQ, a two-step post-training quantization method for vision transformers that significantly reduces quantization errors and improves accuracy by addressing activation and weight quantization errors sequentially.

Contribution

ERQ is a novel two-step PTQ approach that combines reparameterization, ridge regression, and iterative rounding refinement to effectively minimize quantization errors in ViTs.

Findings

ERQ outperforms state-of-the-art GPTQ by 36.81% in accuracy on W3A4 ViT-S.

ERQ achieves superior performance across various ViT variants and tasks.

The method effectively reduces quantization errors, leading to higher model accuracy.

Abstract

Post-training quantization (PTQ) for vision transformers (ViTs) has received increasing attention from both academic and industrial communities due to its minimal data needs and high time efficiency. However, many current methods fail to account for the complex interactions between quantized weights and activations, resulting in significant quantization errors and suboptimal performance. This paper presents ERQ, an innovative two-step PTQ method specifically crafted to reduce quantization errors arising from activation and weight quantization sequentially. The first step, Activation quantization error reduction (Aqer), first applies Reparameterization Initialization aimed at mitigating initial quantization errors in high-variance activations. Then, it further mitigates the errors by formulating a Ridge Regression problem, which updates the weights maintained at full-precision using a closed-form solution. The second step, Weight quantization error reduction (Wqer), first applies Dual Uniform Quantization to handle weights with numerous outliers, which arise from adjustments made during Reparameterization Initialization, thereby reducing initial weight quantization errors. Then, it employs an iterative approach to further tackle the errors. In each iteration, it adopts Rounding Refinement that uses an empirically derived, efficient proxy to refine the rounding directions of quantized weights, complemented by a Ridge Regression solver to reduce the errors. Comprehensive experimental results demonstrate ERQ's superior performance across various ViTs variants and tasks. For example, ERQ surpasses the state-of-the-art GPTQ by a notable 36.81% in accuracy for W3A4 ViT-S. Our codes are available at https://github.com/zysxmu/ERQ.