Adaptive Graph Learning from Spatial Information for Surgical Workflow Anticipation

TL;DR

This paper introduces an adaptive graph learning framework utilizing spatial information from surgical videos to improve anticipation of surgical events, enhancing accuracy over existing methods and supporting better surgical team coordination.

Contribution

The paper presents a novel spatial representation based on bounding boxes, an adaptive graph learning approach for dynamic interactions, and a multi-horizon prediction objective for surgical workflow anticipation.

Findings

Achieved approximately 3% error reduction in surgical phase prediction.

Achieved approximately 9% error reduction in remaining surgical duration prediction.

Demonstrated superior performance on two benchmark datasets.

Abstract

Surgical workflow anticipation is the task of predicting the timing of relevant surgical events from live video data, which is critical in Robotic-Assisted Surgery (RAS). Accurate predictions require the use of spatial information to model surgical interactions. However, current methods focus solely on surgical instruments, assume static interactions between instruments, and only anticipate surgical events within a fixed time horizon. To address these challenges, we propose an adaptive graph learning framework for surgical workflow anticipation based on a novel spatial representation, featuring three key innovations. First, we introduce a new representation of spatial information based on bounding boxes of surgical instruments and targets, including their detection confidence levels. These are trained on additional annotations we provide for two benchmark datasets. Second, we design an adaptive graph learning method to capture dynamic interactions. Third, we develop a multi-horizon objective that balances learning objectives for different time horizons, allowing for unconstrained predictions. Evaluations on two benchmarks reveal superior performance in short-to-mid-term anticipation, with an error reduction of approximately 3% for surgical phase anticipation and 9% for remaining surgical duration anticipation. These performance improvements demonstrate the effectiveness of our method and highlight its potential for enhancing preparation and coordination within the RAS team. This can improve surgical safety and the efficiency of operating room usage.