Towards Holistic Surgical Scene Graph
Jongmin Shin, Enki Cho, Ka Young Kim, Jung Yong Kim, Seong Tae Kim, and Namkee Oh

TL;DR
This paper introduces a new dataset and graph-based method for comprehensive surgical scene understanding, emphasizing the importance of representing tool-action-target relationships and hand identity for improved analysis.
Contribution
It presents the Endoscapes-SG201 dataset with detailed annotations and the SSG-Com method to incorporate critical scene elements into surgical graph representations.
Findings
Enhanced scene understanding through integrated graph components
Improved performance on safety assessment and action recognition tasks
Demonstrated the significance of tool-action-target and hand identity in surgical graphs
Abstract
Surgical scene understanding is crucial for computer-assisted intervention systems, requiring visual comprehension of surgical scenes that involves diverse elements such as surgical tools, anatomical structures, and their interactions. To effectively represent the complex information in surgical scenes, graph-based approaches have been explored to structurally model surgical entities and their relationships. Previous surgical scene graph studies have demonstrated the feasibility of representing surgical scenes using graphs. However, certain aspects of surgical scenes-such as diverse combinations of tool-action-target and the identity of the hand operating the tool-remain underexplored in graph-based representations, despite their importance. To incorporate these aspects into graph representations, we propose Endoscapes-SG201 dataset, which includes annotations for tool-action-target…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Image Retrieval and Classification Techniques
