TexPro: Text-guided PBR Texturing with Procedural Material Modeling

TL;DR

TexPro is a novel method that generates diverse, high-fidelity, procedural textures for 3D meshes from text prompts, enabling physically-based rendering and relighting, surpassing existing methods.

Contribution

The paper introduces TexPro, which combines text-to-image models with differentiable procedural materials for high-quality, relightable 3D texturing, including a novel material agent for improved matching.

Findings

Outperforms existing state-of-the-art methods.

Enables relighting of textured 3D models.

Produces diverse, physically-based textures from text prompts.

Abstract

In this paper, we present TexPro, a novel method for high-fidelity material generation for input 3D meshes given text prompts. Unlike existing text-conditioned texture generation methods that typically generate RGB textures with baked lighting, TexPro is able to produce diverse texture maps via procedural material modeling, which enables physically-based rendering, relighting, and additional benefits inherent to procedural materials. Specifically, we first generate multi-view reference images given the input textual prompt by employing the latest text-to-image model. We then derive texture maps through rendering-based optimization with recent differentiable procedural materials. To this end, we design several techniques to handle the misalignment between the generated multi-view images and 3D meshes, and introduce a novel material agent that enhances material classification and matching by exploring both part-level understanding and object-aware material reasoning. Experiments demonstrate the superiority of the proposed method over existing SOTAs, and its capability of relighting.