Intelligent Control of Robotic X-ray Devices using a Language-promptable Digital Twin

TL;DR

This paper presents a novel system that uses language prompts and a digital twin to control robotic X-ray devices, enabling flexible, autonomous imaging tasks based on natural language commands.

Contribution

It introduces a language-aligned foundation model for X-ray image segmentation integrated with a digital twin for autonomous control of robotic C-arm X-ray systems.

Findings

Achieved 84% success rate in verbal command-based localization and collimation.

Localized 35 structures within 51.68 mm accuracy across arbitrary orientations.

Demonstrated autonomous visualization and patient-specific viewfinding capabilities.

Abstract

Natural language offers a convenient, flexible interface for controlling robotic C-arm X-ray systems, making advanced functionality and controls accessible. However, enabling language interfaces requires specialized AI models that interpret X-ray images to create a semantic representation for reasoning. The fixed outputs of such AI models limit the functionality of language controls. Incorporating flexible, language-aligned AI models prompted through language enables more versatile interfaces for diverse tasks and procedures. Using a language-aligned foundation model for X-ray image segmentation, our system continually updates a patient digital twin based on sparse reconstructions of desired anatomical structures. This supports autonomous capabilities such as visualization, patient-specific viewfinding, and automatic collimation from novel viewpoints, enabling commands 'Focus in on the lower lumbar vertebrae.' In a cadaver study, users visualized, localized, and collimated structures across the torso using verbal commands, achieving 84% end-to-end success. Post hoc analysis of randomly oriented images showed our patient digital twin could localize 35 commonly requested structures to within 51.68 mm, enabling localization and isolation from arbitrary orientations. Our results demonstrate how intelligent robotic X-ray systems can incorporate physicians' expressed intent directly. While existing foundation models for intra-operative X-ray analysis exhibit failure modes, as they improve, they can facilitate highly flexible, intelligent robotic C-arms.