Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2

TL;DR

This paper introduces SLM-SAM 2, a novel architecture with short- and long-term memory banks that significantly improves the accuracy and efficiency of annotating volumetric medical images by propagating masks more reliably across slices.

Contribution

The paper proposes SLM-SAM 2, an innovative model that enhances medical image annotation by integrating separate memory modules, addressing error propagation issues in existing foundation models.

Findings

Outperforms SAM 2 with 0.14 and 0.10 Dice improvements.

Reduces mask correction time by approximately 60.6%.

Demonstrates robustness across MRI, CT, and ultrasound datasets.

Abstract

Manual annotation of volumetric medical images, such as magnetic resonance imaging (MRI) and computed tomography (CT), is a labor-intensive and time-consuming process. Recent advancements in foundation models for video object segmentation, such as Segment Anything Model 2 (SAM 2), offer a potential opportunity to significantly speed up the annotation process by manually annotating one or a few slices and then propagating target masks across the entire volume. However, the performance of SAM 2 in this context varies. Our experiments show that relying on a single memory bank and attention module is prone to error propagation, particularly at boundary regions where the target is present in the previous slice but absent in the current one. To address this problem, we propose Short-Long Memory SAM 2 (SLM-SAM 2), a novel architecture that integrates distinct short-term and long-term memory banks with separate attention modules to improve segmentation accuracy. We evaluate SLM-SAM 2 on four public datasets covering organs, bones, and muscles across MRI, CT, and ultrasound videos. We show that the proposed method markedly outperforms the default SAM 2, achieving an average Dice Similarity Coefficient improvement of 0.14 and 0.10 in the scenarios when 5 volumes and 1 volume are available for the initial adaptation, respectively. SLM-SAM 2 also exhibits stronger resistance to over-propagation, reducing the time required to correct propagated masks by 60.575% per volume compared to SAM 2, making a notable step toward more accurate automated annotation of medical images for segmentation model development.