MatDecompSDF: High-Fidelity 3D Shape and PBR Material Decomposition from Multi-View Images

TL;DR

MatDecompSDF is a comprehensive framework that jointly recovers detailed 3D shapes and physically-based material properties from multi-view images, enabling realistic rendering and editing of digital assets.

Contribution

The paper introduces a novel end-to-end neural framework combining SDF, material prediction, and lighting estimation with physical priors for high-fidelity 3D decomposition from images.

Findings

Outperforms state-of-the-art in geometric accuracy and material fidelity

Produces editable, relightable 3D assets suitable for graphics pipelines

Effective on both synthetic and real-world datasets

Abstract

We present MatDecompSDF, a novel framework for recovering high-fidelity 3D shapes and decomposing their physically-based material properties from multi-view images. The core challenge of inverse rendering lies in the ill-posed disentanglement of geometry, materials, and illumination from 2D observations. Our method addresses this by jointly optimizing three neural components: a neural Signed Distance Function (SDF) to represent complex geometry, a spatially-varying neural field for predicting PBR material parameters (albedo, roughness, metallic), and an MLP-based model for capturing unknown environmental lighting. The key to our approach is a physically-based differentiable rendering layer that connects these 3D properties to the input images, allowing for end-to-end optimization. We introduce a set of carefully designed physical priors and geometric regularizations, including a material smoothness loss and an Eikonal loss, to effectively constrain the problem and achieve robust decomposition. Extensive experiments on both synthetic and real-world datasets (e.g., DTU) demonstrate that MatDecompSDF surpasses state-of-the-art methods in geometric accuracy, material fidelity, and novel view synthesis. Crucially, our method produces editable and relightable assets that can be seamlessly integrated into standard graphics pipelines, validating its practical utility for digital content creation.