Towards Holistic Surgical Scene Graph

TL;DR

This paper introduces a new dataset and graph-based method for comprehensive surgical scene understanding, emphasizing the importance of representing tool-action-target relationships and hand identity for improved analysis.

Contribution

It presents the Endoscapes-SG201 dataset with detailed annotations and the SSG-Com method to incorporate critical scene elements into surgical graph representations.

Findings

Enhanced scene understanding through integrated graph components

Improved performance on safety assessment and action recognition tasks

Demonstrated the significance of tool-action-target and hand identity in surgical graphs

Abstract

Surgical scene understanding is crucial for computer-assisted intervention systems, requiring visual comprehension of surgical scenes that involves diverse elements such as surgical tools, anatomical structures, and their interactions. To effectively represent the complex information in surgical scenes, graph-based approaches have been explored to structurally model surgical entities and their relationships. Previous surgical scene graph studies have demonstrated the feasibility of representing surgical scenes using graphs. However, certain aspects of surgical scenes-such as diverse combinations of tool-action-target and the identity of the hand operating the tool-remain underexplored in graph-based representations, despite their importance. To incorporate these aspects into graph representations, we propose Endoscapes-SG201 dataset, which includes annotations for tool-action-target combinations and hand identity. We also introduce SSG-Com, a graph-based method designed to learn and represent these critical elements. Through experiments on downstream tasks such as critical view of safety assessment and action triplet recognition, we demonstrated the importance of integrating these essential scene graph components, highlighting their significant contribution to surgical scene understanding. The code and dataset are available at https://github.com/ailab-kyunghee/SSG-Com