An evaluation of SVBRDF Prediction from Generative Image Models for Appearance Modeling of 3D Scenes

TL;DR

This paper evaluates the use of generative image models for predicting SVBRDF maps to enhance appearance modeling of 3D scenes, highlighting the potential and challenges of integrating these technologies.

Contribution

It provides an analysis of SVBRDF prediction from generative images, comparing neural architectures, and discusses the benefits and limitations of this approach for 3D scene appearance modeling.

Findings

Standard UNet performs competitively with complex models.

Generated images can improve SVBRDF estimation over photographs.

Single-view predictions may cause multiview incoherence.

Abstract

Digital content creation is experiencing a profound change with the advent of deep generative models. For texturing, conditional image generators now allow the synthesis of realistic RGB images of a 3D scene that align with the geometry of that scene. For appearance modeling, SVBRDF prediction networks recover material parameters from RGB images. Combining these technologies allows us to quickly generate SVBRDF maps for multiple views of a 3D scene, which can be merged to form a SVBRDF texture atlas of that scene. In this paper, we analyze the challenges and opportunities for SVBRDF prediction in the context of such a fast appearance modeling pipeline. On the one hand, single-view SVBRDF predictions might suffer from multiview incoherence and yield inconsistent texture atlases. On the other hand, generated RGB images, and the different modalities on which they are conditioned, can provide additional information for SVBRDF estimation compared to photographs. We compare neural architectures and conditions to identify designs that achieve high accuracy and coherence. We find that, surprisingly, a standard UNet is competitive with more complex designs. Project page: http://repo-sam.inria.fr/nerphys/svbrdf-evaluation