UMat: Uncertainty-Aware Single Image High Resolution Material Capture

TL;DR

This paper introduces UMat, a novel deep learning approach for high-resolution material capture from a single diffuse image, incorporating uncertainty modeling to improve reliability and generalization.

Contribution

The work presents a new generative network with attention and a U-Net discriminator for single-image material digitization, and introduces a framework for quantifying model uncertainty.

Findings

Outperforms previous methods in detail and resolution.

Effectively models uncertainty to improve trustworthiness.

Enables active learning for dataset enhancement.

Abstract

We propose a learning-based method to recover normals, specularity, and roughness from a single diffuse image of a material, using microgeometry appearance as our primary cue. Previous methods that work on single images tend to produce over-smooth outputs with artifacts, operate at limited resolution, or train one model per class with little room for generalization. Previous methods that work on single images tend to produce over-smooth outputs with artifacts, operate at limited resolution, or train one model per class with little room for generalization. In contrast, in this work, we propose a novel capture approach that leverages a generative network with attention and a U-Net discriminator, which shows outstanding performance integrating global information at reduced computational complexity. We showcase the performance of our method with a real dataset of digitized textile materials and show that a commodity flatbed scanner can produce the type of diffuse illumination required as input to our method. Additionally, because the problem might be illposed -- more than a single diffuse image might be needed to disambiguate the specular reflection -- or because the training dataset is not representative enough of the real distribution, we propose a novel framework to quantify the model's confidence about its prediction at test time. Our method is the first one to deal with the problem of modeling uncertainty in material digitization, increasing the trustworthiness of the process and enabling more intelligent strategies for dataset creation, as we demonstrate with an active learning experiment.