SurgPLAN++: Universal Surgical Phase Localization Network for Online and Offline Inference

TL;DR

SurgPLAN++ is a universal surgical phase localization network that improves both online and offline surgical video analysis by predicting phase segments globally across entire videos, utilizing phase proposals and data augmentation.

Contribution

The paper introduces SurgPLAN++, a novel framework that enhances surgical phase recognition by integrating global phase localization and a unified approach for online and offline inference.

Findings

Outperforms state-of-the-art methods in both online and offline modes.

Utilizes phase proposals and data augmentation for improved accuracy.

Achieves significant improvements in surgical video analysis performance.

Abstract

Surgical phase recognition is critical for assisting surgeons in understanding surgical videos. Existing studies focused more on online surgical phase recognition, by leveraging preceding frames to predict the current frame. Despite great progress, they formulated the task as a series of frame-wise classification, which resulted in a lack of global context of the entire procedure and incoherent predictions. Moreover, besides online analysis, accurate offline surgical phase recognition is also in significant clinical need for retrospective analysis, and existing online algorithms do not fully analyze the entire video, thereby limiting accuracy in offline analysis. To overcome these challenges and enhance both online and offline inference capabilities, we propose a universal Surgical Phase Localization Network, named SurgPLAN++, with the principle of temporal detection. To ensure a global understanding of the surgical procedure, we devise a phase localization strategy for SurgPLAN++ to predict phase segments across the entire video through phase proposals. For online analysis, to generate high-quality phase proposals, SurgPLAN++ incorporates a data augmentation strategy to extend the streaming video into a pseudo-complete video through mirroring, center-duplication, and down-sampling. For offline analysis, SurgPLAN++ capitalizes on its global phase prediction framework to continuously refine preceding predictions during each online inference step, thereby significantly improving the accuracy of phase recognition. We perform extensive experiments to validate the effectiveness, and our SurgPLAN++ achieves remarkable performance in both online and offline modes, which outperforms state-of-the-art methods. The source code is available at https://github.com/franciszchen/SurgPLAN-Plus.