Collaborative Control for Geometry-Conditioned PBR Image Generation

TL;DR

This paper introduces a novel method for directly modeling PBR image distributions to improve the accuracy of physically-based rendering in 3D content generation, overcoming limitations of RGB-based approaches.

Contribution

It proposes a new cross-network communication paradigm to train a PBR model linked to a frozen RGB model, maintaining general performance and compatibility.

Findings

Successfully models PBR images directly, reducing photometric inaccuracies.

Maintains compatibility with existing RGB models and adapters.

Addresses data scarcity and high-dimensionality challenges in PBR generation.

Abstract

Graphics pipelines require physically-based rendering (PBR) materials, yet current 3D content generation approaches are built on RGB models. We propose to model the PBR image distribution directly, avoiding photometric inaccuracies in RGB generation and the inherent ambiguity in extracting PBR from RGB. As existing paradigms for cross-modal fine-tuning are not suited for PBR generation due to both a lack of data and the high dimensionality of the output modalities, we propose to train a new PBR model that is tightly linked to a frozen RGB model using a novel cross-network communication paradigm. As the base RGB model is fully frozen, the proposed method retains its general performance and remains compatible with e.g. IPAdapters for that base model.