SemiVT-Surge: Semi-Supervised Video Transformer for Surgical Phase Recognition

TL;DR

This paper introduces SemiVT-Surge, a semi-supervised video transformer model that leverages unlabeled surgical videos with pseudo-labeling and contrastive learning to improve surgical phase recognition accuracy.

Contribution

It presents a novel semi-supervised learning framework for surgical video analysis combining temporal consistency and contrastive learning within a transformer architecture.

Findings

Achieves 4.9% accuracy increase on RAMIE dataset.

Performs comparably to fully supervised methods with only 25% labeled data on Cholec80.

Establishes a new benchmark for semi-supervised surgical phase recognition.

Abstract

Accurate surgical phase recognition is crucial for computer-assisted interventions and surgical video analysis. Annotating long surgical videos is labor-intensive, driving research toward leveraging unlabeled data for strong performance with minimal annotations. Although self-supervised learning has gained popularity by enabling large-scale pretraining followed by fine-tuning on small labeled subsets, semi-supervised approaches remain largely underexplored in the surgical domain. In this work, we propose a video transformer-based model with a robust pseudo-labeling framework. Our method incorporates temporal consistency regularization for unlabeled data and contrastive learning with class prototypes, which leverages both labeled data and pseudo-labels to refine the feature space. Through extensive experiments on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public Cholec80 dataset, we demonstrate the effectiveness of our approach. By incorporating unlabeled data, we achieve state-of-the-art performance on RAMIE with a 4.9% accuracy increase and obtain comparable results to full supervision while using only 1/4 of the labeled data on Cholec80. Our findings establish a strong benchmark for semi-supervised surgical phase recognition, paving the way for future research in this domain.