MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets

TL;DR

MaterialSeg3D introduces a novel framework that infers and segments 3D object materials from 2D semantic priors, overcoming limitations of existing 2D-based methods for more accurate and relightable 3D asset creation.

Contribution

It proposes a new 3D material inference method using a semantic prior model, a UV stack fusion technique, and a new dataset for training and evaluation.

Findings

Effective 3D material segmentation demonstrated through experiments

Outperforms existing methods in relightability and material accuracy

New dataset (MIO) enhances semantic and material understanding

Abstract

Driven by powerful image diffusion models, recent research has achieved the automatic creation of 3D objects from textual or visual guidance. By performing score distillation sampling (SDS) iteratively across different views, these methods succeed in lifting 2D generative prior to the 3D space. However, such a 2D generative image prior bakes the effect of illumination and shadow into the texture. As a result, material maps optimized by SDS inevitably involve spurious correlated components. The absence of precise material definition makes it infeasible to relight the generated assets reasonably in novel scenes, which limits their application in downstream scenarios. In contrast, humans can effortlessly circumvent this ambiguity by deducing the material of the object from its appearance and semantics. Motivated by this insight, we propose MaterialSeg3D, a 3D asset material generation framework to infer underlying material from the 2D semantic prior. Based on such a prior model, we devise a mechanism to parse material in 3D space. We maintain a UV stack, each map of which is unprojected from a specific viewpoint. After traversing all viewpoints, we fuse the stack through a weighted voting scheme and then employ region unification to ensure the coherence of the object parts. To fuel the learning of semantics prior, we collect a material dataset, named Materialized Individual Objects (MIO), which features abundant images, diverse categories, and accurate annotations. Extensive quantitative and qualitative experiments demonstrate the effectiveness of our method.