3-D Material Style Transfer for Reconstructing Unknown Appearance in Complex Natural Materials

TL;DR

This paper introduces a 3-D material style transfer method that reconstructs the appearance of complex natural materials with intricate patterns by transferring material properties from an exemplar to a target object, enabling applications like artifact restoration.

Contribution

The work presents a novel approach using spherical harmonics and property distribution transfer to accurately reconstruct complex material appearances without relying on shape correspondence.

Findings

More accurate color pattern replication than existing methods

Effective reconstruction of faded or extinct material appearances

Applications demonstrated in artifact restoration and synthetic texture generation

Abstract

We propose a 3-D material style transfer framework for reconstructing invisible (or faded) appearance properties in complex natural materials. Our algorithm addresses the technical challenge of transferring appearance properties from one object to another of the same material when both objects have intricate, noncorresponding color patterns. Eggshells, exoskeletons, and minerals, for example, have patterns composed of highly randomized layers of organic and inorganic compounds. These materials pose a challenge as the distribution of compounds that determine surface color changes from object to object and within local pattern regions. Our solution adapts appearance observations from a material property distribution in an exemplar to the material property distribution of a target object to reconstruct its unknown appearance. We use measured reflectance in 3-D bispectral textures to record changing material property distributions. Our novel implementation of spherical harmonics uses principles from chemistry and biology to learn relationships between color (hue and saturation) and material composition and concentration in an exemplar. The encoded relationships are transformed to the property distribution of a target for color recovery and material assignment. Quantitative and qualitative evaluation methods show that we replicate color patterns more accurately than methods that only rely on shape correspondences and coarse-level perceptual differences. We demonstrate applications of our work for reconstructing color in extinct fossils, restoring faded artifacts and generating synthetic textures.