PhotoScene: Photorealistic Material and Lighting Transfer for Indoor Scenes
Yu-Ying Yeh, Zhengqin Li, Yannick Hold-Geoffroy, Rui Zhu, Zexiang Xu,, Milo\v{s} Ha\v{s}an, Kalyan Sunkavalli, Manmohan Chandraker

TL;DR
PhotoScene creates photorealistic 3D indoor scene models with high-quality materials and lighting, enabling realistic re-rendering from arbitrary viewpoints and lighting conditions, based on input images and CAD geometry.
Contribution
It introduces a novel framework that combines procedural material graphs and differentiable rendering to produce photorealistic, relightable 3D scenes from images and CAD models.
Findings
Reconstructs high-quality, fully relightable 3D scenes.
Achieves photorealistic material and lighting transfer.
Demonstrates effectiveness on various datasets.
Abstract
Most indoor 3D scene reconstruction methods focus on recovering 3D geometry and scene layout. In this work, we go beyond this to propose PhotoScene, a framework that takes input image(s) of a scene along with approximately aligned CAD geometry (either reconstructed automatically or manually specified) and builds a photorealistic digital twin with high-quality materials and similar lighting. We model scene materials using procedural material graphs; such graphs represent photorealistic and resolution-independent materials. We optimize the parameters of these graphs and their texture scale and rotation, as well as the scene lighting to best match the input image via a differentiable rendering layer. We evaluate our technique on objects and layout reconstructions from ScanNet, SUN RGB-D and stock photographs, and demonstrate that our method reconstructs high-quality, fully relightable 3D…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · 3D Surveying and Cultural Heritage · Robotics and Sensor-Based Localization
