A DirectX-Based DICOM Viewer for Multi-User Surgical Planning in Augmented Reality

TL;DR

This paper presents a real-time holographic DICOM viewer for HoloLens 2, enabling immersive 3D medical imaging visualization and multi-user interaction for surgical planning, leveraging DirectX and OpenXR APIs.

Contribution

The paper introduces a novel holographic DICOM viewer prototype for HoloLens 2 with real-time rendering, multi-user interaction, and device-agnostic deployment capabilities.

Findings

Real-time 3D volume rendering on HoloLens 2

Multiple user interaction methods implemented

Preliminary positive qualitative evaluation

Abstract

Preoperative medical imaging is an essential part of surgical planning. The data from medical imaging devices, such as CT and MRI scanners, consist of stacks of 2D images in DICOM format. Conversely, advances in 3D data visualization provide further information by assembling cross-sections into 3D volumetric datasets. As Microsoft unveiled the HoloLens 2 (HL2), which is considered one of the best Mixed Reality (XR) headsets in the market, it promised to enhance visualization in 3D by providing an immersive experience to users. This paper introduces a prototype holographic XR DICOM Viewer for the 3D visualization of DICOM image sets on HL2 for surgical planning. We first developed a standalone graphical C++ engine using the native DirectX11 API and HLSL shaders. Based on that, the prototype further applies the OpenXR API for potential deployment on a wide range of devices from vendors across the XR spectrum. With native access to the device, our prototype unravels the limitation of hardware capabilities on HL2 for 3D volume rendering and interaction. Moreover, smartphones can act as input devices to provide another user interaction method by connecting to our server. In this paper, we present a holographic DICOM viewer for the HoloLens 2 and contribute (i) a prototype that renders the DICOM image stacks in real-time on HL2, (ii) three types of user interactions in XR, and (iii) a preliminary qualitative evaluation of our prototype.