SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction

TL;DR

This paper presents SANGRIA, an innovative end-to-end framework that generates and optimizes surgical scene graphs using unsupervised learning and foundation models, significantly improving surgical workflow recognition accuracy.

Contribution

It introduces a novel unsupervised scene graph generation method with spatiotemporal optimization for surgical videos, reducing reliance on dense annotations.

Findings

Outperforms state-of-the-art on CATARACTS dataset by 8% accuracy

Achieves 10% higher F1 score in surgical workflow recognition

Effectively uses weak labels for scene graph optimization

Abstract

Graph-based holistic scene representations facilitate surgical workflow understanding and have recently demonstrated significant success. However, this task is often hindered by the limited availability of densely annotated surgical scene data. In this work, we introduce an end-to-end framework for the generation and optimization of surgical scene graphs on a downstream task. Our approach leverages the flexibility of graph-based spectral clustering and the generalization capability of foundation models to generate unsupervised scene graphs with learnable properties. We reinforce the initial spatial graph with sparse temporal connections using local matches between consecutive frames to predict temporally consistent clusters across a temporal neighborhood. By jointly optimizing the spatiotemporal relations and node features of the dynamic scene graph with the downstream task of phase segmentation, we address the costly and annotation-burdensome task of semantic scene comprehension and scene graph generation in surgical videos using only weak surgical phase labels. Further, by incorporating effective intermediate scene representation disentanglement steps within the pipeline, our solution outperforms the SOTA on the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow recognition