DEAP-3DSAM: Decoder Enhanced and Auto Prompt SAM for 3D Medical Image Segmentation

TL;DR

DEAP-3DSAM introduces a novel decoder and auto prompt mechanism to improve 3D medical image segmentation with SAM, achieving state-of-the-art results without manual prompts.

Contribution

The paper presents a feature-enhanced decoder and dual attention auto prompt system, addressing spatial feature loss and manual prompt reliance in 3D medical segmentation with SAM.

Findings

Achieves state-of-the-art performance on four abdominal tumor datasets.

Outperforms existing manual prompt methods.

Validated effectiveness through ablation studies.

Abstract

The Segment Anything Model (SAM) has recently demonstrated significant potential in medical image segmentation. Although SAM is primarily trained on 2D images, attempts have been made to apply it to 3D medical image segmentation. However, the pseudo 3D processing used to adapt SAM results in spatial feature loss, limiting its performance. Additionally, most SAM-based methods still rely on manual prompts, which are challenging to implement in real-world scenarios and require extensive external expert knowledge. To address these limitations, we introduce the Decoder Enhanced and Auto Prompt SAM (DEAP-3DSAM) to tackle these limitations. Specifically, we propose a Feature Enhanced Decoder that fuses the original image features with rich and detailed spatial information to enhance spatial features. We also design a Dual Attention Prompter to automatically obtain prompt information through Spatial Attention and Channel Attention. We conduct comprehensive experiments on four public abdominal tumor segmentation datasets. The results indicate that our DEAP-3DSAM achieves state-of-the-art performance in 3D image segmentation, outperforming or matching existing manual prompt methods. Furthermore, both quantitative and qualitative ablation studies confirm the effectiveness of our proposed modules.