ReflectanceFusion: Diffusion-based text to SVBRDF Generation

TL;DR

ReflectanceFusion is a diffusion-based neural model that generates high-fidelity, editable SVBRDF maps from textual descriptions, enabling versatile and accurate material synthesis with physical parameter control.

Contribution

The paper introduces a novel tandem neural approach combining stable diffusion and a specialized UNet for high-quality, controllable text-to-SVBRDF generation, trained on a large synthetic dataset.

Findings

Achieves superior accuracy compared to existing models like Text2Mat.

Enables generation of multiple SVBRDFs from a single text prompt.

Supports editing of physical parameters such as roughness and specularity.

Abstract

We introduce Reflectance Diffusion, a new neural text-to-texture model capable of generating high-fidelity SVBRDF maps from textual descriptions. Our method leverages a tandem neural approach, consisting of two modules, to accurately model the distribution of spatially varying reflectance as described by text prompts. Initially, we employ a pre-trained stable diffusion 2 model to generate a latent representation that informs the overall shape of the material and serves as our backbone model. Then, our ReflectanceUNet enables fine-tuning control over the material's physical appearance and generates SVBRDF maps. ReflectanceUNet module is trained on an extensive dataset comprising approximately 200,000 synthetic spatially varying materials. Our generative SVBRDF diffusion model allows for the synthesis of multiple SVBRDF estimates from a single textual input, offering users the possibility to choose the output that best aligns with their requirements. We illustrate our method's versatility by generating SVBRDF maps from a range of textual descriptions, both specific and broad. Our ReflectanceUNet model can integrate optional physical parameters, such as roughness and specularity, enhancing customization. When the backbone module is fixed, the ReflectanceUNet module refines the material, allowing direct edits to its physical attributes. Comparative evaluations demonstrate that ReflectanceFusion achieves better accuracy than existing text-to-material models, such as Text2Mat, while also providing the benefits of editable and relightable SVBRDF maps.