InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective

TL;DR

InfoSAM introduces an information-theoretic fine-tuning method for the Segment Anything Model, effectively transferring pre-trained knowledge to enhance performance in specialized domains.

Contribution

It proposes a novel mutual information-based distillation framework for parameter-efficient fine-tuning of SAM, preserving domain-invariant knowledge.

Findings

Improves SAM performance on real-world benchmarks.

Enhances domain adaptation in specialized tasks.

Outperforms existing fine-tuning methods.

Abstract

The Segment Anything Model (SAM), a vision foundation model, exhibits impressive zero-shot capabilities in general tasks but struggles in specialized domains. Parameter-efficient fine-tuning (PEFT) is a promising approach to unleash the potential of SAM in novel scenarios. However, existing PEFT methods for SAM neglect the domain-invariant relations encoded in the pre-trained model. To bridge this gap, we propose InfoSAM, an information-theoretic approach that enhances SAM fine-tuning by distilling and preserving its pre-trained segmentation knowledge. Specifically, we formulate the knowledge transfer process as two novel mutual information-based objectives: (i) to compress the domain-invariant relation extracted from pre-trained SAM, excluding pseudo-invariant information as possible, and (ii) to maximize mutual information between the relational knowledge learned by the teacher (pre-trained SAM) and the student (fine-tuned model). The proposed InfoSAM establishes a robust distillation framework for PEFT of SAM. Extensive experiments across diverse benchmarks validate InfoSAM's effectiveness in improving SAM family's performance on real-world tasks, demonstrating its adaptability and superiority in handling specialized scenarios.