Intelligent Virtual Sonographer (IVS): Enhancing Physician-Robot-Patient Communication

TL;DR

This paper presents an innovative virtual agent in XR that improves communication and control in robotic ultrasound procedures, enhancing efficiency, patient experience, and trust through LLM-powered dialogue and robotic control integration.

Contribution

It introduces the first integrated IVS system that bridges physician, robot, and patient communication in real-time using LLMs and XR technology.

Findings

Enhanced communication clarity between physician and patient.

Improved patient acceptance of robotic ultrasound.

Real-time control and transparency of robotic actions.

Abstract

The advancement and maturity of large language models (LLMs) and robotics have unlocked vast potential for human-computer interaction, particularly in the field of robotic ultrasound. While existing research primarily focuses on either patient-robot or physician-robot interaction, the role of an intelligent virtual sonographer (IVS) bridging physician-robot-patient communication remains underexplored. This work introduces a conversational virtual agent in Extended Reality (XR) that facilitates real-time interaction between physicians, a robotic ultrasound system(RUS), and patients. The IVS agent communicates with physicians in a professional manner while offering empathetic explanations and reassurance to patients. Furthermore, it actively controls the RUS by executing physician commands and transparently relays these actions to the patient. By integrating LLM-powered dialogue with speech-to-text, text-to-speech, and robotic control, our system enhances the efficiency, clarity, and accessibility of robotic ultrasound acquisition. This work constitutes a first step toward understanding how IVS can bridge communication gaps in physician-robot-patient interaction, providing more control and therefore trust into physician-robot interaction while improving patient experience and acceptance of robotic ultrasound.