Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Model

TL;DR

Mix-QSAM introduces a mixed-precision post-training quantization framework for the Segment Anything Model, leveraging layer importance and inter-layer dependencies to optimize bit-width allocation, resulting in improved accuracy and efficiency on segmentation and detection tasks.

Contribution

It proposes a novel mixed-precision PTQ method for SAM using importance and synergy metrics, formulated as an IQP problem for optimal bit-width distribution.

Findings

Achieves up to 20% higher average precision with 6-bit and 4-bit mixed-precision settings.

Outperforms existing PTQ methods on segmentation and detection tasks.

Maintains computational efficiency while improving accuracy.

Abstract

The Segment Anything Model (SAM) is a popular vision foundation model; however, its high computational and memory demands make deployment on resource-constrained devices challenging. While Post-Training Quantization (PTQ) is a practical approach for reducing computational overhead, existing PTQ methods rely on fixed bit-width quantization, leading to suboptimal accuracy and efficiency. To address this limitation, we propose Mix-QSAM, a mixed-precision PTQ framework for SAM. First, we introduce a layer-wise importance score, derived using Kullback-Leibler (KL) divergence, to quantify each layer's contribution to the model's output. Second, we introduce cross-layer synergy, a novel metric based on causal mutual information, to capture dependencies between adjacent layers. This ensures that highly interdependent layers maintain similar bit-widths, preventing abrupt precision mismatches that degrade feature propagation and numerical stability. Using these metrics, we formulate an Integer Quadratic Programming (IQP) problem to determine optimal bit-width allocation under model size and bit-operation constraints, assigning higher precision to critical layers while minimizing bit-width in less influential layers. Experimental results demonstrate that Mix-QSAM consistently outperforms existing PTQ methods on instance segmentation and object detection tasks, achieving up to 20% higher average precision under 6-bit and 4-bit mixed-precision settings, while maintaining computational efficiency.