LitAR: Visually Coherent Lighting for Mobile Augmented Reality

TL;DR

LitAR is a novel framework for mobile augmented reality that provides realistic, visually-coherent environment lighting by combining spatial variance-aware reconstruction, noise-tolerant data capturing, and real-time rendering techniques.

Contribution

LitAR introduces a comprehensive lighting reconstruction framework that improves visual coherence and efficiency for mobile AR by addressing spatial variance, data quality, and real-time rendering challenges.

Findings

Enables reflective rendering with correct color tones in mobile AR.

Uses two-field lighting reconstruction for spatial variance handling.

Employs real-time environment map rendering techniques to improve efficiency.

Abstract

An accurate understanding of omnidirectional environment lighting is crucial for high-quality virtual object rendering in mobile augmented reality (AR). In particular, to support reflective rendering, existing methods have leveraged deep learning models to estimate or have used physical light probes to capture physical lighting, typically represented in the form of an environment map. However, these methods often fail to provide visually coherent details or require additional setups. For example, the commercial framework ARKit uses a convolutional neural network that can generate realistic environment maps; however the corresponding reflective rendering might not match the physical environments. In this work, we present the design and implementation of a lighting reconstruction framework called LitAR that enables realistic and visually-coherent rendering. LitAR addresses several challenges of supporting lighting information for mobile AR. First, to address the spatial variance problem, LitAR uses two-field lighting reconstruction to divide the lighting reconstruction task into the spatial variance-aware near-field reconstruction and the directional-aware far-field reconstruction. The corresponding environment map allows reflective rendering with correct color tones. Second, LitAR uses two noise-tolerant data capturing policies to ensure data quality, namely guided bootstrapped movement and motion-based automatic capturing. Third, to handle the mismatch between the mobile computation capability and the high computation requirement of lighting reconstruction, LitAR employs two novel real-time environment map rendering techniques called multi-resolution projection and anchor extrapolation. These two techniques effectively remove the need of time-consuming mesh reconstruction while maintaining visual quality.