Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control

TL;DR

This paper introduces a novel diffusion model for PBR Text-to-Texture that directly generates multi-view consistent full PBR stacks, addressing a key challenge in multi-view image synthesis.

Contribution

It presents the first diffusion model capable of directly producing full PBR stacks with multi-view consistency for Text-to-Texture tasks.

Findings

Model achieves multi-view consistency in PBR texture generation

Demonstrates effectiveness through ablation studies

Applicable to practical multi-view texture synthesis

Abstract

Multi-view consistency remains a challenge for image diffusion models. Even within the Text-to-Texture problem, where perfect geometric correspondences are known a priori, many methods fail to yield aligned predictions across views, necessitating non-trivial fusion methods to incorporate the results onto the original mesh. We explore this issue for a Collaborative Control workflow specifically in PBR Text-to-Texture. Collaborative Control directly models PBR image probability distributions, including normal bump maps; to our knowledge, the only diffusion model to directly output full PBR stacks. We discuss the design decisions involved in making this model multi-view consistent, and demonstrate the effectiveness of our approach in ablation studies, as well as practical applications.