Mixed Non-linear Quantization for Vision Transformers

TL;DR

This paper introduces a mixed non-linear quantization approach for Vision Transformers that assigns different quantization methods to each non-linear operation based on layer-wise sensitivity, improving model compression without significant accuracy loss.

Contribution

It proposes a novel mixed non-linear quantization method that optimally assigns quantization techniques to each non-linear layer based on sensitivity, outperforming existing methods.

Findings

Outperforms I-BERT, FQ-ViT, and I-ViT in 8-bit and 6-bit settings.

Achieves an average of 0.6%p and 19.6%p improvement.

Maintains accuracy with limited training time.

Abstract

The majority of quantization methods have been proposed to reduce the model size of Vision Transformers, yet most of them have overlooked the quantization of non-linear operations. Only a few works have addressed quantization for non-linear operations, but they applied a single quantization method across all non-linear operations. We believe that this can be further improved by employing a different quantization method for each non-linear operation. Therefore, to assign the most error-minimizing quantization method from the known methods to each non-linear layer, we propose a mixed non-linear quantization that considers layer-wise quantization sensitivity measured by SQNR difference metric. The results show that our method outperforms I-BERT, FQ-ViT, and I-ViT in both 8-bit and 6-bit settings for ViT, DeiT, and Swin models by an average of 0.6%p and 19.6%p, respectively. Our method outperforms I-BERT and I-ViT by 0.6%p and 20.8%p, respectively, when training time is limited. We plan to release our code at https://gitlab.com/ones-ai/mixed-non-linear-quantization.