MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects

TL;DR

MatPredict introduces a comprehensive dataset and benchmark for learning material properties of indoor objects from images, combining synthetic data with diverse materials, lighting, and camera setups to improve robotic perception.

Contribution

The paper presents a new dataset and benchmark for inferring material properties from images, integrating synthetic objects with varied materials and scene conditions for improved model training.

Findings

Neural network models can infer material properties with varying accuracy.

Diverse lighting and camera setups impact model performance.

The dataset enables large-scale simulation for realistic training.

Abstract

Determining material properties from camera images can expand the ability to identify complex objects in indoor environments, which is valuable for consumer robotics applications. To support this, we introduce MatPredict, a dataset that combines the high-quality synthetic objects from Replica dataset with MatSynth dataset's material properties classes - to create objects with diverse material properties. We select 3D meshes of specific foreground objects and render them with different material properties. In total, we generate \textbf{18} commonly occurring objects with \textbf{14} different materials. We showcase how we provide variability in terms of lighting and camera placement for these objects. Next, we provide a benchmark for inferring material properties from visual images using these perturbed models in the scene, discussing the specific neural network models involved and their performance based on different image comparison metrics. By accurately simulating light interactions with different materials, we can enhance realism, which is crucial for training models effectively through large-scale simulations. This research aims to revolutionize perception in consumer robotics. The dataset is provided \href{https://huggingface.co/datasets/UMTRI/MatPredict}{here} and the code is provided \href{https://github.com/arpan-kusari/MatPredict}{here}.