QMamba: Post-Training Quantization for Vision State Space Models

TL;DR

QMamba introduces a novel post-training quantization framework for vision state space models, effectively reducing computational costs and outperforming existing methods on ImageNet classification.

Contribution

It is among the first PTQ frameworks specifically designed for vision SSMs, utilizing analysis of activation distributions to improve quantization accuracy.

Findings

QMamba surpasses existing PTQ methods by 21.0% on ImageNet with 4-bit activations.

The framework effectively quantizes discrete parameters and hidden states in vision SSMs.

Extensive experiments validate QMamba's superior performance across multiple model sizes and architectures.

Abstract

State Space Models (SSMs), as key components of Mamaba, have gained increasing attention for vision models recently, thanks to their efficient long sequence modeling capability. Given the computational cost of deploying SSMs on resource-limited edge devices, Post-Training Quantization (PTQ) is a technique with the potential for efficient deployment of SSMs. In this work, we propose QMamba, one of the first PTQ frameworks to our knowledge, designed for vision SSMs based on the analysis of the activation distributions in SSMs. We reveal that the distribution of discrete parameters exhibits long-tailed skewness and the distribution of the hidden state sequence exhibits highly dynamic variations. Correspondingly, we design Long-tailed Skewness Quantization (LtSQ) to quantize discrete parameters and Temporal Group Quantization (TGQ) to quantize hidden states, which reduces the quantization errors. Extensive experiments demonstrate that QMamba outperforms advanced PTQ methods on vision models across multiple model sizes and architectures. Notably, QMamba surpasses existing methods by 21.0% on ImageNet classification with 4-bit activations.