ReMeDI: Refined Memory for Disambiguation of Identities with SAM3 in Surgical Segmentation

TL;DR

ReMeDI-SAM3 enhances surgical instrument segmentation in endoscopy by integrating memory filtering, interpolation, and re-identification to improve occlusion handling and identity recovery, achieving significant performance gains without additional training.

Contribution

It introduces a training-free extension to SAM3 that effectively manages occlusions and identity disambiguation in surgical videos through novel memory and re-identification modules.

Findings

Achieved around 6% mcIoU improvement on EndoVis17.

Outperformed prior training-based methods.

Effective zero-shot performance in surgical scene segmentation.

Abstract

Accurate surgical instrument segmentation in endoscopy is crucial for computer-assisted interventions, yet remains challenging due to frequent occlusions, rapid motion, and long-term instrument re-entry. While SAM3 provides a powerful spatio-temporal framework for video object segmentation, its performance in surgical scenes is limited by indiscriminate memory updates, fixed memory capacity, and weak identity recovery after occlusions. We propose ReMeDI-SAM3, a training-free extension of SAM3, that addresses these limitations through three components: (i) relevance-aware memory filtering with a dedicated occlusion-aware memory for storing pre-occlusion frames, (ii) a piecewise interpolation scheme that expands effective memory capacity, and (iii) a feature-based re-identification module with temporal voting for reliable post-occlusion identity disambiguation. Together, these components mitigate error accumulation and enable reliable recovery after occlusions. Evaluations on EndoVis17, EndoVis18 and CholecSeg8k under a zero-shot setting show mcIoU improvements of around 5.8\%, 8\%, and 2\% respectively, over vanilla SAM3, outperforming even prior training-based approaches.