PTQ4SAM: Post-Training Quantization for Segment Anything

TL;DR

PTQ4SAM introduces a post-training quantization framework for the Segment Anything Model, effectively reducing memory and computation costs while maintaining high accuracy across various vision tasks.

Contribution

The paper proposes novel bimodal distribution transformation and adaptive softmax quantization strategies tailored for SAM, enabling efficient low-bit quantization without significant accuracy loss.

Findings

Achieves lossless accuracy at 6-bit quantization for SAM-L in instance segmentation.

Realizes approximately 3.9× acceleration with minimal accuracy drop.

Demonstrates superior performance across multiple vision tasks and datasets.

Abstract

Segment Anything Model (SAM) has achieved impressive performance in many computer vision tasks. However, as a large-scale model, the immense memory and computation costs hinder its practical deployment. In this paper, we propose a post-training quantization (PTQ) framework for Segment Anything Model, namely PTQ4SAM. First, we investigate the inherent bottleneck of SAM quantization attributed to the bimodal distribution in post-Key-Linear activations. We analyze its characteristics from both per-tensor and per-channel perspectives, and propose a Bimodal Integration strategy, which utilizes a mathematically equivalent sign operation to transform the bimodal distribution into a relatively easy-quantized normal distribution offline. Second, SAM encompasses diverse attention mechanisms (i.e., self-attention and two-way cross-attention), resulting in substantial variations in the post-Softmax distributions. Therefore, we introduce an Adaptive Granularity Quantization for Softmax through searching the optimal power-of-two base, which is hardware-friendly. Extensive experimental results across various vision tasks (instance segmentation, semantic segmentation and object detection), datasets and model variants show the superiority of PTQ4SAM. For example, when quantizing SAM-L to 6-bit, we achieve lossless accuracy for instance segmentation, about 0.5\% drop with theoretical 3.9$\times$ acceleration. The code is available at \url{https://github.com/chengtao-lv/PTQ4SAM}.