TraSeTR: Track-to-Segment Transformer with Contrastive Query for Instance-level Instrument Segmentation in Robotic Surgery

TL;DR

TraSeTR is a novel transformer-based approach that improves instrument segmentation in robotic surgery by integrating tracking cues and contrastive learning, achieving state-of-the-art results on multiple datasets.

Contribution

Introduces TraSeTR, a Track-to-Segment Transformer that leverages temporal tracking and contrastive query learning for enhanced instrument segmentation in surgical videos.

Findings

Achieves state-of-the-art segmentation accuracy on three public datasets.

Effectively handles large temporal variations with contrastive query learning.

Demonstrates superior performance over previous methods in instrument type discrimination.

Abstract

Surgical instrument segmentation -- in general a pixel classification task -- is fundamentally crucial for promoting cognitive intelligence in robot-assisted surgery (RAS). However, previous methods are struggling with discriminating instrument types and instances. To address the above issues, we explore a mask classification paradigm that produces per-segment predictions. We propose TraSeTR, a novel Track-to-Segment Transformer that wisely exploits tracking cues to assist surgical instrument segmentation. TraSeTR jointly reasons about the instrument type, location, and identity with instance-level predictions i.e., a set of class-bbox-mask pairs, by decoding query embeddings. Specifically, we introduce the prior query that encoded with previous temporal knowledge, to transfer tracking signals to current instances via identity matching. A contrastive query learning strategy is further applied to reshape the query feature space, which greatly alleviates the tracking difficulty caused by large temporal variations. The effectiveness of our method is demonstrated with state-of-the-art instrument type segmentation results on three public datasets, including two RAS benchmarks from EndoVis Challenges and one cataract surgery dataset CaDIS.