4D-OR: Semantic Scene Graphs for OR Domain Modeling

TL;DR

This paper introduces a novel 4D semantic scene graph representation for modeling surgical environments, validated with a new dataset and neural network pipeline, enabling automated understanding and clinical role prediction in operating rooms.

Contribution

It presents the first use of semantic scene graphs for OR modeling, along with a new dataset and an end-to-end neural network for scene graph generation.

Findings

Achieved 0.75 macro F1 in scene graph inference

Achieved 0.85 macro F1 in clinical role prediction

Created the first publicly available 4D surgical scene graph dataset

Abstract

Surgical procedures are conducted in highly complex operating rooms (OR), comprising different actors, devices, and interactions. To date, only medically trained human experts are capable of understanding all the links and interactions in such a demanding environment. This paper aims to bring the community one step closer to automated, holistic and semantic understanding and modeling of OR domain. Towards this goal, for the first time, we propose using semantic scene graphs (SSG) to describe and summarize the surgical scene. The nodes of the scene graphs represent different actors and objects in the room, such as medical staff, patients, and medical equipment, whereas edges are the relationships between them. To validate the possibilities of the proposed representation, we create the first publicly available 4D surgical SSG dataset, 4D-OR, containing ten simulated total knee replacement surgeries recorded with six RGB-D sensors in a realistic OR simulation center. 4D-OR includes 6734 frames and is richly annotated with SSGs, human and object poses, and clinical roles. We propose an end-to-end neural network-based SSG generation pipeline, with a rate of success of 0.75 macro F1, indeed being able to infer semantic reasoning in the OR. We further demonstrate the representation power of our scene graphs by using it for the problem of clinical role prediction, where we achieve 0.85 macro F1. The code and dataset will be made available upon acceptance.