MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspective

TL;DR

MixA-Q introduces a mixed-precision activation quantization method for vision transformers that leverages activation sparsity to enhance efficiency with minimal accuracy loss, applicable to various quantization techniques.

Contribution

It proposes a novel intra-layer sparsity-aware quantization framework with a Two-Branch Swin Block for improved speed and accuracy in vision transformer inference.

Findings

Achieves 1.35x speedup without accuracy loss in PTQ.

Attains 1.25x speedup with no accuracy loss and 1.53x with slight accuracy drop in QAT.

Improves W4A4 model mAP by 0.7%, reducing quantization degradation by 24%.

Abstract

In this paper, we propose MixA-Q, a mixed-precision activation quantization framework that leverages intra-layer activation sparsity (a concept widely explored in activation pruning methods) for efficient inference of quantized window-based vision transformers. For a given uniform-bit quantization configuration, MixA-Q separates the batched window computations within Swin blocks and assigns a lower bit width to the activations of less important windows, improving the trade-off between model performance and efficiency. We introduce a Two-Branch Swin Block that processes activations separately in high- and low-bit precision, enabling seamless integration of our method with most quantization-aware training (QAT) and post-training quantization (PTQ) methods, or with simple modifications. Our experimental evaluations over the COCO dataset demonstrate that MixA-Q achieves a training-free 1.35x computational speedup without accuracy loss in PTQ configuration. With QAT, MixA-Q achieves a lossless 1.25x speedup and a 1.53x speedup with only a 1% mAP drop by incorporating activation pruning. Notably, by reducing the quantization error in important regions, our sparsity-aware quantization adaptation improves the mAP of the quantized W4A4 model (with both weights and activations in 4-bit precision) by 0.7%, reducing quantization degradation by 24%.