Learning Material-Aware Local Descriptors for 3D Shapes

TL;DR

This paper introduces a neural network-based approach for material-aware analysis of 3D shapes, supported by a new dataset and a mesh-aware smoothing method, enabling applications like texturing and retrieval.

Contribution

It presents a novel projective CNN architecture for material descriptor learning from view-based 3D representations and provides a new dataset with part-wise material labels.

Findings

Effective material classification and retrieval demonstrated

New dataset with 3080 shapes and expert labels released

Mesh-aware CRF improves local material prediction consistency

Abstract

Material understanding is critical for design, geometric modeling, and analysis of functional objects. We enable material-aware 3D shape analysis by employing a projective convolutional neural network architecture to learn material- aware descriptors from view-based representations of 3D points for point-wise material classification or material- aware retrieval. Unfortunately, only a small fraction of shapes in 3D repositories are labeled with physical mate- rials, posing a challenge for learning methods. To address this challenge, we crowdsource a dataset of 3080 3D shapes with part-wise material labels. We focus on furniture models which exhibit interesting structure and material variabil- ity. In addition, we also contribute a high-quality expert- labeled benchmark of 115 shapes from Herman-Miller and IKEA for evaluation. We further apply a mesh-aware con- ditional random field, which incorporates rotational and reflective symmetries, to smooth our local material predic- tions across neighboring surface patches. We demonstrate the effectiveness of our learned descriptors for automatic texturing, material-aware retrieval, and physical simulation. The dataset and code will be publicly available.