STTAR: Surgical Tool Tracking using off-the-shelf Augmented Reality Head-Mounted Displays

TL;DR

This paper introduces a framework that leverages off-the-shelf AR HMDs to accurately track surgical tools with retro-reflective markers, eliminating the need for additional hardware and supporting multiple tools simultaneously.

Contribution

The proposed framework enables precise, multi-tool tracking using built-in AR HMD cameras, demonstrating comparable accuracy to existing medical tracking systems without extra components.

Findings

Achieved sub-millimeter accuracy in marker detection and tracking.

Successfully tracked multiple surgical tools simultaneously.

Validated system performance in simulated orthopedic procedures.

Abstract

The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the task performance. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers, such as those used in surgical procedures, without the need to integrate any additional components. This framework is also capable of simultaneously tracking multiple tools. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09 +- 0.06 mm on lateral translation, 0.42 +- 0.32 mm on longitudinal translation, and 0.80 +- 0.39 deg for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, two surgeons and one biomedical researcher were provided with visual navigation, each performing 21 injections. Results from this use case provide comparable accuracy to those reported in the literature for AR-based navigation procedures.