Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

TL;DR

This paper presents a device-agnostic surgical tool tracking framework that fuses multiple sensing modalities and uses multi-view optimization to improve robustness and accuracy of augmented reality guidance during surgeries, even under occlusions.

Contribution

It introduces a novel multi-modal tracking framework with real-time reliability estimation, enhancing AR visualization robustness in dynamic, occlusion-prone surgical environments.

Findings

Improved tracking robustness under occlusions.

Enhanced AR visualization consistency.

Effective multi-sensor data fusion for surgical tracking.

Abstract

Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.