FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation

TL;DR

FIMA-Q introduces a novel post-training quantization method for Vision Transformers that leverages Fisher Information Matrix approximation to significantly improve accuracy, especially under low-bit quantization, without retraining.

Contribution

The paper proposes FIMA-Q, a new PTQ approach for ViTs that uses FIM approximation via DPLR-FIM and establishes a connection between KL divergence and FIM for efficient quantization loss computation.

Findings

Substantially improves accuracy over state-of-the-art PTQ methods.

Effective in low-bit quantization scenarios.

Validated across various vision tasks and ViT architectures.

Abstract

Post-training quantization (PTQ) has stood out as a cost-effective and promising model compression paradigm in recent years, as it avoids computationally intensive model retraining. Nevertheless, current PTQ methods for Vision Transformers (ViTs) still suffer from significant accuracy degradation, especially under low-bit quantization. To address these shortcomings, we analyze the prevailing Hessian-guided quantization loss, and uncover certain limitations of conventional Hessian approximations. By following the block-wise reconstruction framework, we propose a novel PTQ method for ViTs, dubbed FIMA-Q. Specifically, we firstly establish the connection between KL divergence and FIM, which enables fast computation of the quantization loss during reconstruction. We further propose an efficient FIM approximation method, namely DPLR-FIM, by employing the diagonal plus low-rank principle, and formulate the ultimate quantization loss. Our extensive experiments, conducted across various vision tasks with representative ViT-based architectures on public datasets, demonstrate that our method substantially promotes the accuracy compared to the state-of-the-art approaches, especially in the case of low-bit quantization. The source code is available at https://github.com/ShiheWang/FIMA-Q.