Dynamic Scene Graph Representation for Surgical Video

TL;DR

This paper introduces a novel approach using scene graphs and graph convolutional networks to improve surgical workflow recognition from complex surgical videos, enhancing explainability and robustness in clinical applications.

Contribution

The paper proposes a new scene graph representation for surgical videos and demonstrates its effectiveness for workflow recognition using GCNs, with added benefits of interpretability and robustness.

Findings

Scene graphs enable better surgical workflow recognition.

Graph convolutional networks effectively leverage scene graphs.

Enhanced explainability and robustness in surgical decision models.

Abstract

Surgical videos captured from microscopic or endoscopic imaging devices are rich but complex sources of information, depicting different tools and anatomical structures utilized during an extended amount of time. Despite containing crucial workflow information and being commonly recorded in many procedures, usage of surgical videos for automated surgical workflow understanding is still limited. In this work, we exploit scene graphs as a more holistic, semantically meaningful and human-readable way to represent surgical videos while encoding all anatomical structures, tools, and their interactions. To properly evaluate the impact of our solutions, we create a scene graph dataset from semantic segmentations from the CaDIS and CATARACTS datasets. We demonstrate that scene graphs can be leveraged through the use of graph convolutional networks (GCNs) to tackle surgical downstream tasks such as surgical workflow recognition with competitive performance. Moreover, we demonstrate the benefits of surgical scene graphs regarding the explainability and robustness of model decisions, which are crucial in the clinical setting.