InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modeling

TL;DR

InvRGB+L introduces a unified inverse rendering approach that combines RGB and LiDAR data, leveraging LiDAR's spectral intensity cues for improved material estimation and scene reconstruction in complex, dynamic environments.

Contribution

The paper proposes a novel physics-based LiDAR shading model and RGB-LiDAR material consistency losses, enhancing inverse rendering accuracy by integrating LiDAR intensity information.

Findings

Outperforms state-of-the-art methods in urban inverse rendering

Enables relighting and dynamic scene editing

Produces high-quality novel-view RGB and LiDAR renderings

Abstract

We present InvRGB+L, a novel inverse rendering model that reconstructs large, relightable, and dynamic scenes from a single RGB+LiDAR sequence. Conventional inverse graphics methods rely primarily on RGB observations and use LiDAR mainly for geometric information, often resulting in suboptimal material estimates due to visible light interference. We find that LiDAR's intensity values-captured with active illumination in a different spectral range-offer complementary cues for robust material estimation under variable lighting. Inspired by this, InvRGB+L leverages LiDAR intensity cues to overcome challenges inherent in RGB-centric inverse graphics through two key innovations: (1) a novel physics-based LiDAR shading model and (2) RGB-LiDAR material consistency losses. The model produces novel-view RGB and LiDAR renderings of urban and indoor scenes and supports relighting, night simulations, and dynamic object insertions, achieving results that surpass current state-of-the-art methods in both scene-level urban inverse rendering and LiDAR simulation.