eXplainMR: Generating Real-time Textual and Visual eXplanations to Facilitate UltraSonography Learning in MR

TL;DR

eXplainMR is an innovative mixed reality system that enhances ultrasound training by providing real-time visual and textual explanations, interactive feedback, and 3D cues to improve learning efficiency and accessibility.

Contribution

The paper introduces a novel mixed reality tutoring system that combines real-time image segmentation, interactive troubleshooting, and visual cues for ultrasound education.

Findings

Improved trainee understanding of ultrasound views.

Enhanced engagement through interactive explanations.

Potential for low-cost, accessible ultrasound training.

Abstract

eXplainMR is a Mixed Reality tutoring system designed for basic cardiac surface ultrasound training. Trainees wear a head-mounted display (HMD) and hold a controller, mimicking a real ultrasound probe, while treating a desk surface as the patient's body for low-cost and anywhere training. eXplainMR engages trainees with troubleshooting questions and provides automated feedback through four key mechanisms: 1) subgoals that break down tasks into single-movement steps, 2) textual explanations comparing the current incorrect view with the target view, 3) real-time segmentation and annotation of ultrasound images for direct visualization, and 4) the 3D visual cues provide further explanations on the intersection between the slicing plane and anatomies.