Coordinated 2D-3D Visualization of Volumetric Medical Data in XR with Multimodal Interactions

TL;DR

This paper presents an XR-based system for volumetric medical data visualization that integrates multi-layered reconstructions with multimodal interactions, improving spatial understanding and usability for medical exploration.

Contribution

The paper introduces a novel XR system combining coordinated visualization with multimodal interactions, including LLM-enabled voice commands, to enhance medical data exploration.

Findings

Improved task completion times in user study

Enhanced usability metrics with multimodal interactions

Effective integration of 3D models and multi-planar reconstructions

Abstract

Volumetric medical imaging technologies produce detailed 3D representations of anatomical structures. However, effective medical data visualization and exploration pose significant challenges, especially for individuals with limited medical expertise. We introduce a novel XR-based system with two key innovations: (1) a coordinated visualization module integrating Multi-layered Multi-planar Reconstruction with 3D mesh models and (2) a multimodal interaction framework combining hand gestures with LLM-enabled voice commands. We conduct preliminary evaluations, including a 15-participant user study and expert interviews, to demonstrate the system's abilities to enhance spatial understanding and reduce cognitive load. Experimental results show notable improvements in task completion times, usability metrics, and interaction effectiveness enhanced by LLM-driven voice control. While identifying areas for future refinement, our findings highlight the potential of this immersive visualization system to advance medical training and clinical practice. Our demo application and supplemental materials are available for download at: https://osf.io/bpjq5/.