Shape and Reflectance Reconstruction in Uncontrolled Environments by Differentiable Rendering

TL;DR

This paper introduces a differentiable rendering approach for reconstructing 3D geometry and reflectance from multi-view images in uncontrolled environments, enabling photorealistic novel view synthesis.

Contribution

It presents a novel, efficient method that automatically builds a virtual scene matching real-world parameters using differentiable rendering and photometric optimization.

Findings

Achieves accurate reconstruction of complex geometry and reflectance.

Produces high-quality, photorealistic novel views.

Outperforms state-of-the-art methods in view synthesis.

Abstract

Simultaneous reconstruction of geometry and reflectance properties in uncontrolled environments remains a challenging problem. In this paper, we propose an efficient method to reconstruct the scene's 3D geometry and reflectance from multi-view photography using conventional hand-held cameras. Our method automatically builds a virtual scene in a differentiable rendering system that roughly matches the real world's scene parameters, optimized by minimizing photometric objectives alternatingly and stochastically. With the optimal scene parameters evaluated, photo-realistic novel views for various viewing angles and distances can then be generated by our approach. We present the results of captured scenes with complex geometry and various reflection types. Our method also shows superior performance compared to state-of-the-art alternatives in novel view synthesis visually and quantitatively.