IllumiNet: Transferring Illumination from Planar Surfaces to Virtual Objects in Augmented Reality

TL;DR

IllumiNet introduces a learning-based method for transferring illumination from planar surfaces in a single RGB image to virtual objects, enabling realistic AR experiences in diverse environments without reconstructing full scene lighting.

Contribution

The paper proposes a novel approach that directly infers illumination for virtual objects from planar surfaces, improving robustness and applicability in indoor and outdoor scenes.

Findings

Outperforms state-of-the-art in realism and accuracy

Works effectively in both indoor and outdoor environments

Enhances augmented reality experiences with realistic lighting

Abstract

This paper presents an illumination estimation method for virtual objects in real environment by learning. While previous works tackled this problem by reconstructing high dynamic range (HDR) environment maps or the corresponding spherical harmonics, we do not seek to recover the lighting environment of the entire scene. Given a single RGB image, our method directly infers the relit virtual object by transferring the illumination features extracted from planar surfaces in the scene to the desired geometries. Compared to previous works, our approach is more robust as it works in both indoor and outdoor environments with spatially-varying illumination. Experiments and evaluation results show that our approach outperforms the state-of-the-art quantitatively and qualitatively, achieving realistic augmented experience.