Segment anything model 2: an application to 2D and 3D medical images

TL;DR

This paper evaluates SAM 2's effectiveness in segmenting 2D and 3D medical images across various modalities, revealing its comparable performance to SAM in 2D and variable results in 3D, guiding future domain-specific adaptations.

Contribution

The study provides a comprehensive evaluation of SAM 2 on diverse medical imaging datasets, highlighting its capabilities and limitations in 2D and 3D segmentation tasks.

Findings

SAM 2 performs similarly to SAM in 2D segmentation.

Performance varies in 3D segmentation depending on slice selection and propagation methods.

The work offers insights for adapting SAM 2 to medical imaging applications.

Abstract

Segment Anything Model (SAM) has gained significant attention because of its ability to segment various objects in images given a prompt. The recently developed SAM 2 has extended this ability to video inputs. This opens an opportunity to apply SAM to 3D images, one of the fundamental tasks in the medical imaging field. In this paper, we extensively evaluate SAM 2's ability to segment both 2D and 3D medical images by first collecting 21 medical imaging datasets, including surgical videos, common 3D modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) as well as 2D modalities such as X-ray and ultrasound. Two evaluation settings of SAM 2 are considered: (1) multi-frame 3D segmentation, where prompts are provided to one or multiple slice(s) selected from the volume, and (2) single-frame 2D segmentation, where prompts are provided to each slice. The former only applies to videos and 3D modalities, while the latter applies to all datasets. Our results show that SAM 2 exhibits similar performance as SAM under single-frame 2D segmentation, and has variable performance under multi-frame 3D segmentation depending on the choices of slices to annotate, the direction of the propagation, the predictions utilized during the propagation, etc. We believe our work enhances the understanding of SAM 2's behavior in the medical field and provides directions for future work in adapting SAM 2 to this domain. Our code is available at: https://github.com/mazurowski-lab/segment-anything2-medical-evaluation.