Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality

TL;DR

This paper introduces a deep neural network model that estimates complex, real-time environment lighting for mixed reality without prior scene information, enabling more realistic XR experiences.

Contribution

It proposes a novel CNN architecture that accurately predicts environment lighting using spherical harmonics from HDR images, improving real-time mixed reality rendering.

Findings

Achieves an average MSE of 7.85e-04 in lighting coefficient prediction.

Successfully validates the model in diverse mixed reality scenarios.

Provides qualitative relighting results of real-world scenes.

Abstract

The representation of consistent mixed reality (XR) environments requires adequate real and virtual illumination composition in real-time. Estimating the lighting of a real scenario is still a challenge. Due to the ill-posed nature of the problem, classical inverse-rendering techniques tackle the problem for simple lighting setups. However, those assumptions do not satisfy the current state-of-art in computer graphics and XR applications. While many recent works solve the problem using machine learning techniques to estimate the environment light and scene's materials, most of them are limited to geometry or previous knowledge. This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images. We show in the experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of \num{7.85e-04} when comparing SH lighting coefficients. We validate our model in a variety of mixed reality scenarios. Furthermore, we present qualitative results comparing relights of real-world scenes.