Hybrelighter: Combining Deep Anisotropic Diffusion and Scene Reconstruction for On-device Real-time Relighting in Mixed Reality

TL;DR

Hybrelighter is a real-time on-device relighting method for mixed reality that combines scene reconstruction, anisotropic diffusion, and filter techniques to produce accurate lighting effects at high speeds.

Contribution

The paper introduces a novel integration of scene understanding, anisotropic diffusion, and filtering for real-time, accurate MR scene relighting on edge devices.

Findings

Achieves up to 100 fps performance.

Provides more accurate relighting than existing methods.

Effectively corrects scene scanning inaccuracies.

Abstract

Mixed Reality scene relighting, where virtual changes to lighting conditions realistically interact with physical objects, producing authentic illumination and shadows, can be used in a variety of applications. One such application in real estate could be visualizing a room at different times of day and placing virtual light fixtures. Existing deep learning-based relighting techniques typically exceed the real-time performance capabilities of current MR devices. On the other hand, scene understanding methods, such as on-device scene reconstruction, often yield inaccurate results due to scanning limitations, in turn affecting relighting quality. Finally, simpler 2D image filter-based approaches cannot represent complex geometry and shadows. We introduce a novel method to integrate image segmentation, with lighting propagation via anisotropic diffusion on top of basic scene understanding, and the computational simplicity of filter-based techniques. Our approach corrects on-device scanning inaccuracies, delivering visually appealing and accurate relighting effects in real-time on edge devices, achieving speeds as high as 100 fps. We show a direct comparison between our method and the industry standard, and present a practical demonstration of our method in the aforementioned real estate example.