Neural Photometry-guided Visual Attribute Transfer

TL;DR

This paper introduces a deep learning method for transferring visual material attributes across images, robust to illumination and deformation, enabling interactive material editing with minimal training data.

Contribution

The method leverages a supervised image translation framework that generalizes from single-material training, supporting various visual attributes and robust to illumination changes.

Findings

Effective transfer of material attributes across different images.

Robustness to illumination variations and affine deformations.

Single-material training suffices for generalization to similar materials.

Abstract

We present a deep learning-based method for propagating spatially-varying visual material attributes (e.g. texture maps or image stylizations) to larger samples of the same or similar materials. For training, we leverage images of the material taken under multiple illuminations and a dedicated data augmentation policy, making the transfer robust to novel illumination conditions and affine deformations. Our model relies on a supervised image-to-image translation framework and is agnostic to the transferred domain; we showcase a semantic segmentation, a normal map, and a stylization. Following an image analogies approach, the method only requires the training data to contain the same visual structures as the input guidance. Our approach works at interactive rates, making it suitable for material edit applications. We thoroughly evaluate our learning methodology in a controlled setup providing quantitative measures of performance. Last, we demonstrate that training the model on a single material is enough to generalize to materials of the same type without the need for massive datasets.