Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks

TL;DR

This paper introduces a CNN-based method for modeling spatially varying surface reflectance from a single photograph, utilizing self-augmentation to reduce the need for extensive labeled training data.

Contribution

The authors propose a self-augmentation training process that leverages unlabeled images to improve SVBRDF modeling from minimal labeled data.

Findings

Effective modeling of SVBRDFs for various materials

Self-augmentation improves network accuracy

Validated on wood, metals, and plastics

Abstract

We present a convolutional neural network (CNN) based solution for modeling physically plausible spatially varying surface reflectance functions (SVBRDF) from a single photograph of a planar material sample under unknown natural illumination. Gathering a sufficiently large set of labeled training pairs consisting of photographs of SVBRDF samples and corresponding reflectance parameters, is a difficult and arduous process. To reduce the amount of required labeled training data, we propose to leverage the appearance information embedded in unlabeled images of spatially varying materials to self-augment the training process. Starting from an initial approximative network obtained from a small set of labeled training pairs, we estimate provisional model parameters for each unlabeled training exemplar. Given this provisional reflectance estimate, we then synthesize a novel temporary labeled training pair by rendering the exact corresponding image under a new lighting condition. After refining the network using these additional training samples, we re-estimate the provisional model parameters for the unlabeled data and repeat the self-augmentation process until convergence. We demonstrate the efficacy of the proposed network structure on spatially varying wood, metals, and plastics, as well as thoroughly validate the effectiveness of the self-augmentation training process.