Deep SVBRDF Estimation on Real Materials

TL;DR

This paper demonstrates that training deep learning models for SVBRDF estimation on synthetic data alone is insufficient for real-world accuracy, and proposes finetuning and per-material optimization to improve results on real materials.

Contribution

It introduces a new real-material dataset and a novel deep learning architecture, showing that adaptation techniques significantly enhance SVBRDF estimation on real data.

Findings

Finetuning on real data improves accuracy.

Per-material optimization further enhances results.

Synthetic training alone is inadequate for real-world applications.

Abstract

Recent work has demonstrated that deep learning approaches can successfully be used to recover accurate estimates of the spatially-varying BRDF (SVBRDF) of a surface from as little as a single image. Closer inspection reveals, however, that most approaches in the literature are trained purely on synthetic data, which, while diverse and realistic, is often not representative of the richness of the real world. In this paper, we show that training such networks exclusively on synthetic data is insufficient to achieve adequate results when tested on real data. Our analysis leverages a new dataset of real materials obtained with a novel portable multi-light capture apparatus. Through an extensive series of experiments and with the use of a novel deep learning architecture, we explore two strategies for improving results on real data: finetuning, and a per-material optimization procedure. We show that adapting network weights to real data is of critical importance, resulting in an approach which significantly outperforms previous methods for SVBRDF estimation on real materials. Dataset and code are available at https://lvsn.github.io/real-svbrdf