STTAR: Surgical Tool Tracking using off-the-shelf Augmented Reality Head-Mounted Displays
Alejandro Martin-Gomez, Haowei Li, Tianyu Song, Sheng Yang, and Guangzhi Wang, Hui Ding, Nassir Navab, Zhe Zhao, Mehran, Armand

TL;DR
This paper introduces a framework that leverages off-the-shelf AR HMDs to accurately track surgical tools with retro-reflective markers, eliminating the need for additional hardware and supporting multiple tools simultaneously.
Contribution
The proposed framework enables precise, multi-tool tracking using built-in AR HMD cameras, demonstrating comparable accuracy to existing medical tracking systems without extra components.
Findings
Achieved sub-millimeter accuracy in marker detection and tracking.
Successfully tracked multiple surgical tools simultaneously.
Validated system performance in simulated orthopedic procedures.
Abstract
The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the task performance. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers, such as those used in surgical procedures, without the need to integrate any additional components. This…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAugmented Reality Applications · Surgical Simulation and Training · Anatomy and Medical Technology
