RefSAM3D: Adapting SAM with Cross-modal Reference for 3D Medical Image Segmentation

TL;DR

RefSAM3D extends the Segment Anything Model to 3D medical images by incorporating a 3D adapter, cross-modal prompts, and hierarchical attention, significantly improving segmentation accuracy in volumetric medical data.

Contribution

This work introduces a novel adaptation of SAM for 3D medical imaging, including a 3D image adapter and cross-modal prompt generation, enabling effective segmentation of complex anatomical structures.

Findings

Outperforms state-of-the-art segmentation methods on multiple datasets.

Effectively captures multi-scale information with hierarchical attention.

Achieves higher accuracy and consistency in 3D medical image segmentation.

Abstract

The Segment Anything Model (SAM), originally built on a 2D Vision Transformer (ViT), excels at capturing global patterns in 2D natural images but struggles with 3D medical imaging modalities like CT and MRI. These modalities require capturing spatial information in volumetric space for tasks such as organ segmentation and tumor quantification. To address this challenge, we introduce RefSAM3D, which adapts SAM for 3D medical imaging by incorporating a 3D image adapter and cross-modal reference prompt generation. Our approach modifies the visual encoder to handle 3D inputs and enhances the mask decoder for direct 3D mask generation. We also integrate textual prompts to improve segmentation accuracy and consistency in complex anatomical scenarios. By employing a hierarchical attention mechanism, our model effectively captures and integrates information across different scales. Extensive evaluations on multiple medical imaging datasets demonstrate the superior performance of RefSAM3D over state-of-the-art methods. Our contributions advance the application of SAM in accurately segmenting complex anatomical structures in medical imaging.