Realistic Volume Rendering with Environment-Synced Illumination in Mixed Reality

TL;DR

This paper introduces a new mixed reality visualization framework that enhances realistic volume rendering by estimating environment illumination and applying spatio-temporal denoising, improving visual quality and user perception.

Contribution

The paper presents a novel MR visualization framework with efficient environment illumination estimation and a new denoising algorithm for improved realistic volume rendering.

Findings

Enhanced visual quality of DVR in MR using spatio-temporal denoising.

Supports immersive and intuitive perception in MR volume visualization.

Outperforms existing MR solutions in visual realism and stability.

Abstract

Interactive volume visualization using a mixed reality (MR) system helps provide users with an intuitive spatial perception of volumetric data. Due to sophisticated requirements of user interaction and vision when using MR head-mounted display (HMD) devices, the conflict between the realisticness and efficiency of direct volume rendering (DVR) is yet to be resolved. In this paper, a new MR visualization framework that supports interactive realistic DVR is proposed. An efficient illumination estimation method is used to identify the high dynamic range (HDR) environment illumination captured using a panorama camera. To improve the visual quality of Monte Carlo-based DVR, a new spatio-temporal denoising algorithm is designed. Based on a reprojection strategy, it makes full use of temporal coherence between adjacent frames and spatial coherence between the two screens of an HMD to optimize MR rendering quality. Several MR development modules are also developed for related devices to efficiently and stably display the DVR results in an MR HMD. Experimental results demonstrate that our framework can better support immersive and intuitive user perception during MR viewing than existing MR solutions.