MatMart: Material Reconstruction of 3D Objects via Diffusion

TL;DR

MatMart introduces a two-stage diffusion-based framework for 3D object material reconstruction, combining accurate prediction and flexible generation from multiple views with end-to-end training for improved stability.

Contribution

The paper presents MatMart, a novel diffusion model framework that enables scalable, flexible, and stable 3D material reconstruction without pre-trained models.

Findings

Outperforms existing methods in material reconstruction accuracy

Supports arbitrary input views with high scalability

Operates with end-to-end optimization for stability

Abstract

Applying diffusion models to physically-based material estimation and generation has recently gained prominence. In this paper, we propose \ttt, a novel material reconstruction framework for 3D objects, offering the following advantages. First, \ttt\ adopts a two-stage reconstruction, starting with accurate material prediction from inputs and followed by prior-guided material generation for unobserved views, yielding high-fidelity results. Second, by utilizing progressive inference alongside the proposed view-material cross-attention (VMCA), \ttt\ enables reconstruction from an arbitrary number of input images, demonstrating strong scalability and flexibility. Finally, \ttt\ achieves both material prediction and generation capabilities through end-to-end optimization of a single diffusion model, without relying on additional pre-trained models, thereby exhibiting enhanced stability across various types of objects. Extensive experiments demonstrate that \ttt\ achieves superior performance in material reconstruction compared to existing methods.