Q-SAM2: Accurate Quantization for Segment Anything Model 2

TL;DR

Q-SAM2 introduces a novel low-bit quantization method for the Segment Anything Model 2, significantly reducing model size and computational costs while maintaining high segmentation accuracy through innovative calibration and clipping techniques.

Contribution

The paper proposes Variance-Reduced Calibration and Learnable Statistical Clipping, novel methods that improve low-bit quantization performance for SAM2.

Findings

Achieves up to 9.7 percentage points improvement in video segmentation accuracy.

Reduces model size by 8 times compared to BF16 baseline.

Outperforms state-of-the-art quantization schemes in ultra-low 2-bit regime.

Abstract

The Segment Anything Model 2 (SAM2) is a powerful foundation model for promptable segmentation. However, its high computational and memory costs are a major barrier to deployment on resource-constrained devices. In this paper, we present Q-SAM2, an accurate low-bit quantization method that achieves high compression and high fidelity. To address performance degradation arising from challenging weight and activation distributions during quantization, Q-SAM2 introduces two novel contributions: Variance-Reduced Calibration (VRC), an initialization method that reduces weight statistical variance by minimizing the Frobenius norm over a small calibration batch; and Learnable Statistical Clipping (LSC), a Quantization-Aware Training (QAT) method that learns momentum-stabilized clipping factors to manage outliers in weights and activations. Comprehensive experiments demonstrate that Q-SAM2 achieves highly accurate inference with substantial efficiency gains, significantly surpassing state-of-the-art general QAT schemes, particularly in the ultra-low 2-bit regime. Specifically, Q-SAM2 achieves an accuracy gain of up to 9.7 ppt in J&F on the video segmentation benchmark and 7.3 ppt in mIoU for instance segmentation over the best competing QAT model, all while achieving an 8x reduction in model size compared to the BF16 baseline.