PTQ4VM: Post-Training Quantization for Visual Mamba

TL;DR

This paper introduces PTQ4VM, a post-training quantization method tailored for Visual Mamba, addressing its unique quantization challenges to enable fast, low-accuracy-loss model deployment on GPUs.

Contribution

It presents the first quantization approach specifically designed for Visual Mamba, with novel strategies like PTS and JLSS to improve quantization efficiency and effectiveness.

Findings

Achieves up to 1.83x GPU speedup with negligible accuracy loss.

Converts pretrained Visual Mamba models in under 15 minutes.

Effectively addresses quantization challenges unique to Visual Mamba.

Abstract

Visual Mamba is an approach that extends the selective space state model, Mamba, to vision tasks. It processes image tokens sequentially in a fixed order, accumulating information to generate outputs. Despite its growing popularity for delivering high-quality outputs at a low computational cost across various tasks, Visual Mamba is highly susceptible to quantization, which makes further performance improvements challenging. Our analysis reveals that the fixed token access order in Visual Mamba introduces unique quantization challenges, which we categorize into three main issues: 1) token-wise variance, 2) channel-wise outliers, and 3) a long tail of activations. To address these challenges, we propose Post-Training Quantization for Visual Mamba (PTQ4VM), which introduces two key strategies: Per-Token Static (PTS) quantization and Joint Learning of Smoothing Scale and Step Size (JLSS). To the our best knowledge, this is the first quantization study on Visual Mamba. PTQ4VM can be applied to various Visual Mamba backbones, converting the pretrained model to a quantized format in under 15 minutes without notable quality degradation. Extensive experiments on large-scale classification and regression tasks demonstrate its effectiveness, achieving up to 1.83x speedup on GPUs with negligible accuracy loss compared to FP16. Our code is available at https://github.com/YoungHyun197/ptq4vm.