Habitat-Matterport 3D Semantics Dataset

TL;DR

The Habitat-Matterport 3D Semantics dataset is a large, richly annotated collection of real-world 3D indoor spaces designed to advance semantic understanding and navigation tasks in robotics and AI.

Contribution

It introduces the largest and most detailed 3D semantic dataset to date, with pixel-accurate object boundaries and extensive annotations, surpassing prior datasets in scale and quality.

Findings

Policies trained on HM3DSEM outperform those trained on previous datasets.

HM3DSEM significantly improves performance in Object Goal Navigation tasks.

Increased participation in the Habitat ObjectNav Challenge indicates community adoption.

Abstract

We present the Habitat-Matterport 3D Semantics (HM3DSEM) dataset. HM3DSEM is the largest dataset of 3D real-world spaces with densely annotated semantics that is currently available to the academic community. It consists of 142,646 object instance annotations across 216 3D spaces and 3,100 rooms within those spaces. The scale, quality, and diversity of object annotations far exceed those of prior datasets. A key difference setting apart HM3DSEM from other datasets is the use of texture information to annotate pixel-accurate object boundaries. We demonstrate the effectiveness of HM3DSEM dataset for the Object Goal Navigation task using different methods. Policies trained using HM3DSEM perform outperform those trained on prior datasets. Introduction of HM3DSEM in the Habitat ObjectNav Challenge lead to an increase in participation from 400 submissions in 2021 to 1022 submissions in 2022.