Automatic Annotations by Augmented Reality‐Enabled Laparoscopic Surgery

TL;DR

This paper introduces a system that automatically generates surgical annotations using augmented reality, reducing the need for manual labeling by medical experts.

Contribution

A novel method for automatic surgical annotation generation using AR-enabled spatial and temporal registration during laparoscopic surgery.

Findings

The AR system can generate structured labels from synchronized multimodal data streams during surgery.

The system can create annotations even when visual input is occluded or out of camera view.

User evaluations confirmed the feasibility and usability of the AR-based annotation system.

Abstract

Accurate labels of surgical procedures such as image segmentations or interaction labels are paramount for many of today's medical image computing tasks. Creating a dataset with these labels requires a great deal of manual work and relies on the involvement of medical experts, which is very time‐consuming and costly. We propose a pathway for the automatic generation of such labels utilizing the spatial and temporal registration between a patient, the anatomical model, tracked surgical instruments, and the surgeon's view of the patient. These requirements for the automatic generation of labels are identical to the requirements of many navigated and augmented reality (AR) enabled surgeries. The AR system, through 3D registration, has the defining ability to accurately overlay real objects with their virtual counterparts. Our approach collects the complete raw data (e.g. video, tracking data, calibrations etc.) that feeds a live laparoscopic AR system for later analysis. By converting these complete recordings of the surgery into different representations, the AR system generates valuable datasets as mere by‐products. Additionally, as our approach does not rely on visual input alone but on additional 3D information, the system can create labels even if the visual input is occluded or a tool interacts with tissue outside of the view of the laparoscopic camera. In this paper, we present a realization of this concept, then evolve this foundational idea into an interactive system that assists users in annotating surgical data. Finally, we gather and analyse feedback from six participants to evaluate the efficacy and user‐friendliness of our system.

We present a method for the automatic generation of surgical annotations by leveraging the spatial and temporal registration inherent to augmented reality (AR)‐enabled surgery. Our system records and synchronizes complete multimodal data streams from live AR procedures to generate structured labels as by‐products of surgical navigation. A prototype implementation and user evaluation demonstrate the feasibility and usability of this concept.

Figures