Rethinking Text-Promptable Surgical Instrument Segmentation with Robust Framework

TL;DR

This paper introduces R-SIS, a new task for surgical instrument segmentation that handles prompt uncertainty and instrument absence, aiming to improve real-world applicability of promptable segmentation models.

Contribution

The paper formulates the R-SIS task, addressing the limitations of existing promptable segmentation methods by accounting for instrument absence and uncertainty in surgical environments.

Findings

Existing methods show high false positives without instrument presence info.

Current evaluation protocols do not reflect real-world surgical scenarios.

Benchmark results highlight the need for robust segmentation under uncertainty.

Abstract

Surgical instrument segmentation is an essential component of computer-assisted and robotic surgery systems. Vision-based segmentation models typically produce outputs limited to a predefined set of instrument categories, which restricts their applicability in interactive systems and robotic task automation. Promptable segmentation methods allow selective predictions based on textual prompts. However, they often rely on the assumption that the instruments present in the scene are already known, and prompts are generated accordingly, limiting their ability to generalize to unseen or dynamically emerging instruments. In practical surgical environments, where instrument existence information is not provided, this assumption does not hold consistently, resulting in false-positive segmentation. To address these limitations, we formulate a new task called Robust text-promptable Surgical Instrument Segmentation (R-SIS). Under this setting, prompts are issued for all candidate categories without access to instrument presence information. R-SIS requires distinguishing which prompts refer to visible instruments and generating masks only when such instruments are explicitly present in the scene. This setting reflects practical conditions where uncertainty in instrument presence is inherent. We evaluate existing segmentation methods under the R-SIS protocol using surgical video datasets and observe substantial false-positive predictions in the absence of ground-truth instruments. These findings demonstrate a mismatch between current evaluation protocols and real-world use cases, and support the need for benchmarks that explicitly account for prompt uncertainty and instrument absence.