HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigation

TL;DR

This paper introduces HM3D-OVON, a large-scale dataset and benchmark for open-vocabulary object goal navigation, enabling models to find objects specified by free-form language in realistic 3D environments.

Contribution

It provides a new dataset and benchmark that expand the semantic scope of ObjectNav to open-vocabulary goals, facilitating research on more flexible, language-driven navigation.

Findings

Open-vocabulary models outperform traditional fixed-category approaches.

HM3D-OVON-trained agents are more robust to noise.

Benchmark encourages development of human-like semantic navigation.

Abstract

We present the Habitat-Matterport 3D Open Vocabulary Object Goal Navigation dataset (HM3D-OVON), a large-scale benchmark that broadens the scope and semantic range of prior Object Goal Navigation (ObjectNav) benchmarks. Leveraging the HM3DSem dataset, HM3D-OVON incorporates over 15k annotated instances of household objects across 379 distinct categories, derived from photo-realistic 3D scans of real-world environments. In contrast to earlier ObjectNav datasets, which limit goal objects to a predefined set of 6-20 categories, HM3D-OVON facilitates the training and evaluation of models with an open-set of goals defined through free-form language at test-time. Through this open-vocabulary formulation, HM3D-OVON encourages progress towards learning visuo-semantic navigation behaviors that are capable of searching for any object specified by text in an open-vocabulary manner. Additionally, we systematically evaluate and compare several different types of approaches on HM3D-OVON. We find that HM3D-OVON can be used to train an open-vocabulary ObjectNav agent that achieves both higher performance and is more robust to localization and actuation noise than the state-of-the-art ObjectNav approach. We hope that our benchmark and baseline results will drive interest in developing embodied agents that can navigate real-world spaces to find household objects specified through free-form language, taking a step towards more flexible and human-like semantic visual navigation. Code and videos available at: naoki.io/ovon.