Efficient Multi-View Inverse Rendering Using a Hybrid Differentiable Rendering Method

TL;DR

This paper presents a hybrid differentiable rendering approach that efficiently reconstructs 3D shape and reflectance from multi-view images, combining approximate and physically-based methods for improved quality and efficiency.

Contribution

The paper introduces a novel hybrid differentiable rendering technique that integrates approximate and physically-based methods for inverse rendering from multi-view images.

Findings

Achieves high-quality 3D reconstructions comparable to state-of-the-art methods.

Demonstrates improved efficiency over existing inverse rendering techniques.

Validates effectiveness on both synthetic and real-world data.

Abstract

Recovering the shape and appearance of real-world objects from natural 2D images is a long-standing and challenging inverse rendering problem. In this paper, we introduce a novel hybrid differentiable rendering method to efficiently reconstruct the 3D geometry and reflectance of a scene from multi-view images captured by conventional hand-held cameras. Our method follows an analysis-by-synthesis approach and consists of two phases. In the initialization phase, we use traditional SfM and MVS methods to reconstruct a virtual scene roughly matching the real scene. Then in the optimization phase, we adopt a hybrid approach to refine the geometry and reflectance, where the geometry is first optimized using an approximate differentiable rendering method, and the reflectance is optimized afterward using a physically-based differentiable rendering method. Our hybrid approach combines the efficiency of approximate methods with the high-quality results of physically-based methods. Extensive experiments on synthetic and real data demonstrate that our method can produce reconstructions with similar or higher quality than state-of-the-art methods while being more efficient.