Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer

TL;DR

This paper introduces Q-ViT, a fully quantized low-bit vision transformer that maintains high accuracy and efficiency through novel modules, enabling deployment on resource-constrained devices with minimal performance loss.

Contribution

The paper proposes an information rectification module and distribution guided distillation to fully quantize ViTs, significantly reducing computation while preserving or improving accuracy.

Findings

Q-ViT achieves 80.9% Top-1 accuracy on ImageNet.

Q-ViT accelerates ViT-S by 6.14x.

Q-ViT surpasses full-precision models in accuracy.

Abstract

The large pre-trained vision transformers (ViTs) have demonstrated remarkable performance on various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. Among the powerful compression approaches, quantization extremely reduces the computation and memory consumption by low-bit parameters and bit-wise operations. However, low-bit ViTs remain largely unexplored and usually suffer from a significant performance drop compared with the real-valued counterparts. In this work, through extensive empirical analysis, we first identify the bottleneck for severe performance drop comes from the information distortion of the low-bit quantized self-attention map. We then develop an information rectification module (IRM) and a distribution guided distillation (DGD) scheme for fully quantized vision transformers (Q-ViT) to effectively eliminate such distortion, leading to a fully quantized ViTs. We evaluate our methods on popular DeiT and Swin backbones. Extensive experimental results show that our method achieves a much better performance than the prior arts. For example, our Q-ViT can theoretically accelerates the ViT-S by 6.14x and achieves about 80.9% Top-1 accuracy, even surpassing the full-precision counterpart by 1.0% on ImageNet dataset. Our codes and models are attached on https://github.com/YanjingLi0202/Q-ViT