TS-SAM: Fine-Tuning Segment-Anything Model for Downstream Tasks

TL;DR

TS-SAM introduces a novel two-stream fine-tuning approach for the Segment-Anything Model, significantly enhancing its performance on downstream segmentation tasks by integrating feature fusion and multi-scale refinement.

Contribution

The paper proposes TS-SAM, a new fine-tuning framework with a lightweight convolutional side adapter and multi-scale modules, bridging the performance gap with domain-specific models.

Findings

TS-SAM outperforms recent SAM-Adapter and SSOM methods.

Achieves competitive results with state-of-the-art domain-specific models.

Validated on ten public datasets across three segmentation tasks.

Abstract

Adapter based fine-tuning has been studied for improving the performance of SAM on downstream tasks. However, there is still a significant performance gap between fine-tuned SAMs and domain-specific models. To reduce the gap, we propose Two-Stream SAM (TS-SAM). On the one hand, inspired by the side network in Parameter-Efficient Fine-Tuning (PEFT), we designed a lightweight Convolutional Side Adapter (CSA), which integrates the powerful features from SAM into side network training for comprehensive feature fusion. On the other hand, in line with the characteristics of segmentation tasks, we designed Multi-scale Refinement Module (MRM) and Feature Fusion Decoder (FFD) to keep both the detailed and semantic features. Extensive experiments on ten public datasets from three tasks demonstrate that TS-SAM not only significantly outperforms the recently proposed SAM-Adapter and SSOM, but achieves competitive performance with the SOTA domain-specific models. Our code is available at: https://github.com/maoyangou147/TS-SAM.