Zero-Shot Surgical Tool Segmentation in Monocular Video Using Segment Anything Model 2

TL;DR

This paper evaluates the zero-shot surgical tool segmentation capabilities of the Segment Anything Model 2 (SAM 2) in diverse surgical videos, highlighting its strengths and limitations in real-world surgical scenarios.

Contribution

The study demonstrates SAM 2's effectiveness in zero-shot surgical tool segmentation and identifies challenges specific to surgical video analysis.

Findings

SAM 2 performs well across different surgical videos

Additional prompts improve segmentation accuracy for new tools

Surgical video challenges affect SAM 2's robustness

Abstract

The Segment Anything Model 2 (SAM 2) is the latest generation foundation model for image and video segmentation. Trained on the expansive Segment Anything Video (SA-V) dataset, which comprises 35.5 million masks across 50.9K videos, SAM 2 advances its predecessor's capabilities by supporting zero-shot segmentation through various prompts (e.g., points, boxes, and masks). Its robust zero-shot performance and efficient memory usage make SAM 2 particularly appealing for surgical tool segmentation in videos, especially given the scarcity of labeled data and the diversity of surgical procedures. In this study, we evaluate the zero-shot video segmentation performance of the SAM 2 model across different types of surgeries, including endoscopy and microscopy. We also assess its performance on videos featuring single and multiple tools of varying lengths to demonstrate SAM 2's applicability and effectiveness in the surgical domain. We found that: 1) SAM 2 demonstrates a strong capability for segmenting various surgical videos; 2) When new tools enter the scene, additional prompts are necessary to maintain segmentation accuracy; and 3) Specific challenges inherent to surgical videos can impact the robustness of SAM 2.