Compress Any Segment Anything Model (SAM)

TL;DR

This paper introduces Birkhoff, a versatile, data-free compression algorithm for SAM models that achieves high compression ratios and fast inference without fine-tuning, suitable for diverse applications.

Contribution

Birkhoff presents a novel data-free compression method using Hyper-Compression and HyperLinear, offering a universal, fast, and faithful compression solution for SAM models.

Findings

Achieves up to 5.17x compression ratio with less than 1% performance loss.

Compresses models within 60 seconds without fine-tuning.

Performs consistently across multiple datasets and SAM variants.

Abstract

Due to the excellent performance in yielding high-quality, zero-shot segmentation, Segment Anything Model (SAM) and its variants have been widely applied in diverse scenarios such as healthcare and intelligent manufacturing. Therefore, effectively compressing SAMs has become an increasingly pressing practical need. In this study, we propose Birkhoff, a novel data-free compression algorithm for SAM and its variants. Unlike quantization, pruning, distillation, and other compression methods, Birkhoff embodies versatility across model types, agility in deployment, faithfulness to the original model, and compactness in model size. Specifically, Birkhoff introduces a novel compression algorithm: Hyper-Compression, whose core principle is to find a dense trajectory to turn a high-dimensional parameter vector into a low-dimensional scalar. Furthermore, Birkhoff designs a dedicated linear layer operator, HyperLinear, to fuse decompression and matrix multiplication to significantly accelerate inference of the compressed SAMs. Extensive experiments on 18 SAMs in the COCO, LVIS, and SA-1B datasets show that Birkhoff performs consistently and competitively in compression time, compression ratio, post-compression performance, and inference speed. For example, Birkhoff can achieve a compression ratio of 5.17x on SAM2-B, with less than 1% performance drop without using any fine-tuning data. Moreover, the compression is finished within 60 seconds for all models.