LACOSTE: Exploiting stereo and temporal contexts for surgical instrument segmentation

TL;DR

LACOSTE introduces a novel model that leverages stereo and temporal contexts, along with specialized modules, to improve surgical instrument segmentation robustness and accuracy in videos, outperforming previous methods.

Contribution

The paper presents a new approach that exploits stereo and temporal information through three modules, including a disparity-guided feature propagation, a stereo-temporal set classifier, and a location-agnostic classifier, for better segmentation.

Findings

Achieves comparable or superior results to state-of-the-art methods.

Validates effectiveness on three public surgical video datasets.

Demonstrates robustness against appearance variations and view changes.

Abstract

Surgical instrument segmentation is instrumental to minimally invasive surgeries and related applications. Most previous methods formulate this task as single-frame-based instance segmentation while ignoring the natural temporal and stereo attributes of a surgical video. As a result, these methods are less robust against the appearance variation through temporal motion and view change. In this work, we propose a novel LACOSTE model that exploits Location-Agnostic COntexts in Stereo and TEmporal images for improved surgical instrument segmentation. Leveraging a query-based segmentation model as core, we design three performance-enhancing modules. Firstly, we design a disparity-guided feature propagation module to enhance depth-aware features explicitly. To generalize well for even only a monocular video, we apply a pseudo stereo scheme to generate complementary right images. Secondly, we propose a stereo-temporal set classifier, which aggregates stereo-temporal contexts in a universal way for making a consolidated prediction and mitigates transient failures. Finally, we propose a location-agnostic classifier to decouple the location bias from mask prediction and enhance the feature semantics. We extensively validate our approach on three public surgical video datasets, including two benchmarks from EndoVis Challenges and one real radical prostatectomy surgery dataset GraSP. Experimental results demonstrate the promising performances of our method, which consistently achieves comparable or favorable results with previous state-of-the-art approaches.