Domain-agnostic weakly supervised surgical instrument segmentation

TL;DR

This paper introduces a new method for segmenting surgical instruments in medical images without needing manual prompts or annotated data.

Contribution

The novel approach uses an anomaly detector with SAM2 to enable domain-agnostic surgical instrument segmentation without user prompts or annotated datasets.

Findings

The method achieves mean Normalized Surface Distances of 53% to 73% across three surgical datasets.

The proposed method is training-free and mask-free, making it suitable for integration into surgical workflows.

Abstract

Recent advancements in visual foundation models open new avenues in the field of surgical instrument segmentation in medical images. Segmentation foundation models provide high segmentation accuracy for objects of interest that are selected via prompts in the form of points, bounding boxes, or text. However, the choice of suitable prompts either requires manual interaction or relies on two-stage pipelines based on supervised, typically domain-specific models. This limits their applicability for domain-agnostic surgical instrument segmentation. We propose a method for surgical instrument segmentation that leverages the power of the segmentation foundation model SAM2 while eliminating the need for a user-defined input prompt or domain-specific annotated datasets. We achieve this by utilizing an anomaly detector generated from non-instrument images to identify instruments as unseen regions and in this way, define a SAM2 input prompt based solely on image-level annotations. For three datasets for surgical instrument segmentation from diverse domains (EndoVis2017, CaDIS, and PASO-SIS), we achieve mean Normalized Surface Distances ranging from \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$${53}{\%}\,\hbox {to}\,{73}{\%}$$\end{document}. This demonstrates the competitiveness of our method compared to alternatives, while its training- and mask-free nature makes it well-suited for surgical workflow integration. By simplifying surgical instrument segmentation, we advance the field of computer-assisted surgery and unlock a wide variety of assistance functions with minimal effort.

The online version contains supplementary material available at 10.1038/s41598-026-43054-1.

Figures