SD-OVON: A Semantics-aware Dataset and Benchmark Generation Pipeline for Open-Vocabulary Object Navigation in Dynamic Scenes

TL;DR

SD-OVON introduces a semantics-aware dataset and benchmark pipeline for open-vocabulary object navigation in dynamic scenes, enabling realistic training and evaluation of navigation agents in complex, manipulatable environments.

Contribution

It provides a novel pipeline utilizing foundation models to generate diverse, photo-realistic dynamic scenes and a new benchmark dataset for open-vocabulary navigation tasks.

Findings

Demonstrated the effectiveness of the datasets and pipeline through baseline evaluations.

Enhanced realism in navigation tasks with dynamic scenes and manipulatable objects.

Public availability of datasets, benchmarks, and source code.

Abstract

We present the Semantics-aware Dataset and Benchmark Generation Pipeline for Open-vocabulary Object Navigation in Dynamic Scenes (SD-OVON). It utilizes pretraining multimodal foundation models to generate infinite unique photo-realistic scene variants that adhere to real-world semantics and daily commonsense for the training and the evaluation of navigation agents, accompanied with a plugin for generating object navigation task episodes compatible to the Habitat simulator. In addition, we offer two pre-generated object navigation task datasets, SD-OVON-3k and SD-OVON-10k, comprising respectively about 3k and 10k episodes of the open-vocabulary object navigation task, derived from the SD-OVON-Scenes dataset with 2.5k photo-realistic scans of real-world environments and the SD-OVON-Objects dataset with 0.9k manually inspected scanned and artist-created manipulatable object models. Unlike prior datasets limited to static environments, SD-OVON covers dynamic scenes and manipulatable objects, facilitating both real-to-sim and sim-to-real robotic applications. This approach enhances the realism of navigation tasks, the training and the evaluation of open-vocabulary object navigation agents in complex settings. To demonstrate the effectiveness of our pipeline and datasets, we propose two baselines and evaluate them along with state-of-the-art baselines on SD-OVON-3k. The datasets, benchmark and source code are publicly available.