A Dense Material Segmentation Dataset for Indoor and Outdoor Scene Parsing

TL;DR

This paper introduces a large-scale dense material segmentation dataset with 3.2 million segments across diverse indoor and outdoor scenes, significantly improving model performance and establishing a new benchmark for scene parsing.

Contribution

The creation of a comprehensive dataset with 23 times more segments and diverse scene coverage, along with a new benchmark for material segmentation.

Findings

Model trained on our dataset outperforms existing models.

Achieved 0.729 per-pixel accuracy on the benchmark.

Dataset covers diverse scenes, objects, and materials.

Abstract

A key algorithm for understanding the world is material segmentation, which assigns a label (metal, glass, etc.) to each pixel. We find that a model trained on existing data underperforms in some settings and propose to address this with a large-scale dataset of 3.2 million dense segments on 44,560 indoor and outdoor images, which is 23x more segments than existing data. Our data covers a more diverse set of scenes, objects, viewpoints and materials, and contains a more fair distribution of skin types. We show that a model trained on our data outperforms a state-of-the-art model across datasets and viewpoints. We propose a large-scale scene parsing benchmark and baseline of 0.729 per-pixel accuracy, 0.585 mean class accuracy and 0.420 mean IoU across 46 materials.