VLM-Guided Iterative Refinement for Surgical Image Segmentation with Foundation Models

TL;DR

This paper introduces IR-SIS, an innovative iterative surgical image segmentation system that uses foundation models and natural language interaction for adaptive refinement, outperforming existing methods.

Contribution

It presents the first language-guided, adaptive refinement framework for surgical image segmentation, integrating foundation models and clinician interaction.

Findings

State-of-the-art performance on benchmark datasets

Effective clinician-in-the-loop refinement

Robustness to out-of-distribution data

Abstract

Surgical image segmentation is essential for robot-assisted surgery and intraoperative guidance. However, existing methods are constrained to predefined categories, produce one-shot predictions without adaptive refinement, and lack mechanisms for clinician interaction. We propose IR-SIS, an iterative refinement system for surgical image segmentation that accepts natural language descriptions. IR-SIS leverages a fine-tuned SAM3 for initial segmentation, employs a Vision-Language Model to detect instruments and assess segmentation quality, and applies an agentic workflow that adaptively selects refinement strategies. The system supports clinician-in-the-loop interaction through natural language feedback. We also construct a multi-granularity language-annotated dataset from EndoVis2017 and EndoVis2018 benchmarks. Experiments demonstrate state-of-the-art performance on both in-domain and out-of-distribution data, with clinician interaction providing additional improvements. Our work establishes the first language-based surgical segmentation framework with adaptive self-refinement capabilities.