SWAG: Long-term Surgical Workflow Prediction with Generative-based Anticipation

TL;DR

SWAG introduces a generative framework for long-term surgical workflow prediction, combining phase recognition and anticipation to improve intraoperative guidance over extended horizons.

Contribution

The paper presents SWAG, a novel generative-based approach that enhances long-term surgical workflow anticipation using class transition embeddings and a combined regression-classification framework.

Findings

Single-pass model with class transition embeddings achieves up to 41.3% F1 score.

Approach competes effectively in phase remaining time regression tasks.

Versatile framework applicable to both recognition and anticipation tasks.

Abstract

While existing approaches excel at recognising current surgical phases, they provide limited foresight and intraoperative guidance into future procedural steps. Similarly, current anticipation methods are constrained to predicting short-term and single events, neglecting the dense, repetitive, and long sequential nature of surgical workflows. To address these needs and limitations, we propose SWAG (Surgical Workflow Anticipative Generation), a framework that combines phase recognition and anticipation using a generative approach. This paper investigates two distinct decoding methods - single-pass (SP) and auto-regressive (AR) - to generate sequences of future surgical phases at minute intervals over long horizons. We propose a novel embedding approach using class transition probabilities to enhance the accuracy of phase anticipation. Additionally, we propose a generative framework using remaining time regression to classification (R2C). SWAG was evaluated on two publicly available datasets, Cholec80 and AutoLaparo21. Our single-pass model with class transition probability embeddings (SP*) achieves 32.1% and 41.3% F1 scores over 20 and 30 minutes on Cholec80 and AutoLaparo21, respectively. Moreover, our approach competes with existing methods on phase remaining time regression, achieving weighted mean absolute errors of 0.32 and 0.48 minutes for 2- and 3-minute horizons. SWAG demonstrates versatility across generative decoding frame works and classification and regression tasks to create temporal continuity between surgical workflow recognition and anticipation. Our method provides steps towards intraoperative surgical workflow generation for anticipation. Project: https://maxboels.com/research/swag.