Neural Inverse Rendering from Propagating Light

TL;DR

This paper introduces a neural inverse rendering system that accurately models complex light transport from multi-view videos, enabling advanced 3D reconstruction, relighting, and measurement decomposition in scenes with propagating light.

Contribution

It presents the first time-resolved neural inverse rendering method that incorporates propagating light, improving accuracy and enabling new scene editing capabilities.

Findings

Achieves state-of-the-art 3D reconstruction with indirect light effects

Enables view synthesis and relighting of scenes with propagating light

Automatically decomposes measurements into direct and indirect components

Abstract

We present the first system for physically based, neural inverse rendering from multi-viewpoint videos of propagating light. Our approach relies on a time-resolved extension of neural radiance caching -- a technique that accelerates inverse rendering by storing infinite-bounce radiance arriving at any point from any direction. The resulting model accurately accounts for direct and indirect light transport effects and, when applied to captured measurements from a flash lidar system, enables state-of-the-art 3D reconstruction in the presence of strong indirect light. Further, we demonstrate view synthesis of propagating light, automatic decomposition of captured measurements into direct and indirect components, as well as novel capabilities such as multi-view time-resolved relighting of captured scenes.