Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIM

TL;DR

This paper introduces O3D-SIM, a 3D semantic instance mapping method that enhances vision-language navigation by integrating instance-level embeddings into 3D point clouds, improving task success rates and object identification.

Contribution

It extends previous 2D instance-level semantic mapping to 3D, leveraging foundational models for robust, detailed environment understanding in navigation tasks.

Findings

Improved success rate in language-guided navigation tasks.

Enhanced ability to identify objects beyond closed-set limitations.

Qualitative improvements in instance clarity and semantic understanding.

Abstract

Humans excel at forming mental maps of their surroundings, equipping them to understand object relationships and navigate based on language queries. Our previous work, SI Maps (Nanwani L, Agarwal A, Jain K, et al. Instance-level semantic maps for vision language navigation. In: 2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE; 2023 Aug.), showed that having instance-level information and the semantic understanding of an environment helps significantly improve performance for language-guided tasks. We extend this instance-level approach to 3D while increasing the pipeline's robustness and improving quantitative and qualitative results. Our method leverages foundational models for object recognition, image segmentation, and feature extraction. We propose a representation that results in a 3D point cloud map with instance-level embeddings, which bring in the semantic understanding that natural language commands can query. Quantitatively, the work improves upon the success rate of language-guided tasks. At the same time, we qualitatively observe the ability to identify instances more clearly and leverage the foundational models and language and image-aligned embeddings to identify objects that, otherwise, a closed-set approach wouldn't be able to identify. Project Page - https://smart-wheelchair-rrc.github.io/o3d-sim-webpage