DreamPBR: Text-driven Generation of High-resolution SVBRDF with Multi-modal Guidance

TL;DR

DreamPBR is a diffusion-based framework that generates high-resolution, diverse, and controllable spatially-varying PBR materials guided by text and multi-modal inputs, overcoming limitations of prior methods.

Contribution

It introduces a novel diffusion model with multi-modal guidance for high-quality, controllable SVBRDF generation, integrating vision-language models and material priors.

Findings

Supports tileable material generation.

Enables diverse and high-quality material creation.

Provides versatile control through multi-modal guidance.

Abstract

Prior material creation methods had limitations in producing diverse results mainly because reconstruction-based methods relied on real-world measurements and generation-based methods were trained on relatively small material datasets. To address these challenges, we propose DreamPBR, a novel diffusion-based generative framework designed to create spatially-varying appearance properties guided by text and multi-modal controls, providing high controllability and diversity in material generation. Key to achieving diverse and high-quality PBR material generation lies in integrating the capabilities of recent large-scale vision-language models trained on billions of text-image pairs, along with material priors derived from hundreds of PBR material samples. We utilize a novel material Latent Diffusion Model (LDM) to establish the mapping between albedo maps and the corresponding latent space. The latent representation is then decoded into full SVBRDF parameter maps using a rendering-aware PBR decoder. Our method supports tileable generation through convolution with circular padding. Furthermore, we introduce a multi-modal guidance module, which includes pixel-aligned guidance, style image guidance, and 3D shape guidance, to enhance the control capabilities of the material LDM. We demonstrate the effectiveness of DreamPBR in material creation, showcasing its versatility and user-friendliness on a wide range of controllable generation and editing applications.