Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

TL;DR

This paper introduces a novel learning-based inverse rendering framework that estimates scene albedo, normals, depth, and complex 3D lighting from a single image, leveraging a volumetric spherical Gaussian lighting model and a physics-based differentiable renderer.

Contribution

It proposes a unified 3D inverse rendering approach with a new volumetric lighting representation and a physics-based renderer, enabling physically accurate predictions without HDR ground-truth lighting.

Findings

Outperforms prior methods quantitatively and qualitatively.

Capable of photorealistic AR object insertion with specular objects.

Does not require ground-truth HDR lighting for training.

Abstract

In this work, we address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. However, indoor scenes contain complex 3D light transport where a 2D representation is insufficient. In this paper, we propose a unified, learning-based inverse rendering framework that formulates 3D spatially-varying lighting. Inspired by classic volume rendering techniques, we propose a novel Volumetric Spherical Gaussian representation for lighting, which parameterizes the exitant radiance of the 3D scene surfaces on a voxel grid. We design a physics based differentiable renderer that utilizes our 3D lighting representation, and formulates the energy-conserving image formation process that enables joint training of all intrinsic properties with the re-rendering constraint. Our model ensures physically correct predictions and avoids the need for ground-truth HDR lighting which is not easily accessible. Experiments show that our method outperforms prior works both quantitatively and qualitatively, and is capable of producing photorealistic results for AR applications such as virtual object insertion even for highly specular objects.