Physics-based Indirect Illumination for Inverse Rendering

TL;DR

This paper introduces a physics-based inverse rendering approach that explicitly models indirect illumination using a refined sphere tracing algorithm and neural networks, enabling more accurate scene reconstruction and relighting.

Contribution

It proposes a novel differentiable physics-based illumination model that improves inverse rendering by explicitly tracing indirect light and integrating it into end-to-end learning.

Findings

Outperforms existing methods on synthetic and real datasets

Enables realistic material editing and relighting

Achieves better accuracy in geometry and illumination estimation

Abstract

We present a physics-based inverse rendering method that learns the illumination, geometry, and materials of a scene from posed multi-view RGB images. To model the illumination of a scene, existing inverse rendering works either completely ignore the indirect illumination or model it by coarse approximations, leading to sub-optimal illumination, geometry, and material prediction of the scene. In this work, we propose a physics-based illumination model that first locates surface points through an efficient refined sphere tracing algorithm, then explicitly traces the incoming indirect lights at each surface point based on reflection. Then, we estimate each identified indirect light through an efficient neural network. Moreover, we utilize the Leibniz's integral rule to resolve non-differentiability in the proposed illumination model caused by boundary lights inspired by differentiable irradiance in computer graphics. As a result, the proposed differentiable illumination model can be learned end-to-end together with geometry and materials estimation. As a side product, our physics-based inverse rendering model also facilitates flexible and realistic material editing as well as relighting. Extensive experiments on synthetic and real-world datasets demonstrate that the proposed method performs favorably against existing inverse rendering methods on novel view synthesis and inverse rendering.