Beyond the Desktop: XR-Driven Segmentation with Meta Quest 3 and MX Ink

TL;DR

This paper presents an XR-based medical image segmentation tool using Meta Quest 3 and MX Ink stylus, demonstrating its usability and potential to reduce cognitive load in clinical workflows.

Contribution

It introduces an immersive XR interface for medical segmentation that combines stylus interaction with real-time 3D rendering, enhancing workflow and usability.

Findings

Achieved SUS score of 66, indicating acceptable usability.

Participants rated controls as highly intuitive (4.1/5).

Reduced cognitive load compared to traditional desktop tools.

Abstract

Medical imaging segmentation is essential in clinical settings for diagnosing diseases, planning surgeries, and other procedures. However, manual annotation is a cumbersome and effortful task. To mitigate these aspects, this study implements and evaluates the usability and clinical applicability of an extended reality (XR)-based segmentation tool for anatomical CT scans, using the Meta Quest 3 headset and Logitech MX Ink stylus. We develop an immersive interface enabling real-time interaction with 2D and 3D medical imaging data in a customizable workspace designed to mitigate workflow fragmentation and cognitive demands inherent to conventional manual segmentation tools. The platform combines stylus-driven annotation, mirroring traditional pen-on-paper workflows, with instant 3D volumetric rendering. A user study with a public craniofacial CT dataset demonstrated the tool's foundational viability, achieving a System Usability Scale (SUS) score of 66, within the expected range for medical applications. Participants highlighted the system's intuitive controls (scoring 4.1/5 for self-descriptiveness on ISONORM metrics) and spatial interaction design, with qualitative feedback highlighting strengths in hybrid 2D/3D navigation and realistic stylus ergonomics. While users identified opportunities to enhance task-specific precision and error management, the platform's core workflow enabled dynamic slice adjustment, reducing cognitive load compared to desktop tools. Results position the XR-stylus paradigm as a promising foundation for immersive segmentation tools, with iterative refinements targeting haptic feedback calibration and workflow personalization to advance adoption in preoperative planning.