Speak, Segment, Track, Navigate: An Interactive System for Video-Guided Skull-Base Surgery

TL;DR

This paper presents a speech-guided, video-based surgical system for skull base procedures that offers real-time guidance without external hardware, improving workflow and accuracy.

Contribution

It introduces a novel speech-interactive framework that performs perception and guidance tasks directly on intraoperative video streams, eliminating the need for external trackers.

Findings

Achieved a mean tool-tip position error of 2.32 mm in experiments.

Completed segmentation and registration within approximately two minutes.

Demonstrated comparable accuracy to commercial optical tracking systems.

Abstract

We introduce a speech-guided embodied agent framework for video-guided skull base surgery that dynamically executes perception and image-guidance tasks in response to surgeon queries. The proposed system integrates natural language interaction with real-time visual perception directly on live intraoperative video streams, thereby enabling surgeons to request computational assistance without disengaging from operative tasks. Unlike conventional image-guided navigation systems that rely on external optical trackers and additional hardware setup, the framework operates purely on intraoperative video. The system begins with interactive segmentation and labeling of the surgical instrument. The segmented instrument is then used as a spatial anchor that is autonomously tracked in the video stream to support downstream workflows, including anatomical segmentation, interactive registration of preoperative 3D models, monocular video-based estimation of the surgical tool pose, and image guidance through real-time anatomical overlays. We evaluate the proposed system in video-guided skull base surgery scenarios and benchmark its tracking performance against a commercially available optical tracking system. Across three experimental trials, the hybrid vision-based method achieved a mean absolute tool-tip position error of 2.32 Plus/Minus 1.10 mm in the camera frame, with inter-frame yaw and pitch propagation discrepancies of 0.18 Plus/Minus 0.25{\deg} and 0.21 Plus/Minus 0.30{\deg}, respectively. The system completes tool segmentation and anatomy registration within approximately two minutes, substantially reducing setup complexity relative to conventional tracking workflows. These results demonstrate that speech-guided embodied agents can provide accurate spatial guidance while improving workflow integration and enabling rapid deployment of video-guided surgical systems.