PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes

TL;DR

PhotoScene creates photorealistic 3D indoor scene models with high-quality materials and lighting, enabling realistic re-rendering from arbitrary viewpoints and lighting conditions, based on input images and CAD geometry.

Contribution

It introduces a novel framework that combines procedural material graphs and differentiable rendering to produce photorealistic, relightable 3D scenes from images and CAD models.

Findings

Reconstructs high-quality, fully relightable 3D scenes.

Achieves photorealistic material and lighting transfer.

Demonstrates effectiveness on various datasets.

Abstract

Most indoor 3D scene reconstruction methods focus on recovering 3D geometry and scene layout. In this work, we go beyond this to propose PhotoScene, a framework that takes input image(s) of a scene along with approximately aligned CAD geometry (either reconstructed automatically or manually specified) and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We optimize the parameters of these graphs and their texture scale and rotation, as well as the scene lighting to best match the input image via a differentiable rendering layer. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D scenes that can be re-rendered under arbitrary viewpoints, zooms and lighting.