Prompt-Free and Efficient SAM2 Adaptation for Biomedical Semantic Segmentation via Dual Adapters

TL;DR

This paper introduces a prompt-free, efficient adaptation framework for biomedical segmentation using SAM2, combining dual adapters and a convolutional positional encoding generator to improve accuracy and reduce computational costs.

Contribution

It presents a novel prompt-free, parameter-efficient fine-tuning method with dual adapters and a positional encoding generator for biomedical segmentation, outperforming existing baselines.

Findings

Improved segmentation accuracy by up to 19.66% over vanilla SAM2.

Reduced computational costs by approximately 87%.

Effective handling of arbitrary aspect ratios with convolutional positional encoding.

Abstract

Segment Anything Model 2 (SAM2) demonstrated impressive zero-shot capabilities on natural images but faces challenges in biomedical segmentation due to significant domain shifts and prompt dependency. To address these limitations, we propose a prompt-free, parameter-efficient fine-tuning framework designed for multi-class segmentation on variable-sized inputs. We introduce a convolutional Positional Encoding Generator to adapt effectively to arbitrary aspect ratios and present a dual-adapter strategy: High-Performance Adapter utilizing deformable convolutions for precise boundary modeling and Lightweight Adapter employing structural re-parameterization to minimize inference latency. Experiments on ISBI 2012, Kvasir-SEG, Synapse, and ACDC datasets demonstrate that our approach significantly outperforms strong adaptation baselines. Specifically, our method improved segmentation accuracy by up to 19.66\% over the vanilla SAM2, while reducing computational costs by approximately 87\% compared to heavyweight medical SAM adaptations, establishing a superior trade-off between accuracy and efficiency.