OpenMap: Instruction Grounding via Open-Vocabulary Visual-Language Mapping

TL;DR

OpenMap is a zero-shot visual-language mapping system that improves instruction grounding for navigation by combining structural-semantic constraints and large language model assistance, outperforming existing methods in complex 3D environments.

Contribution

The paper introduces OpenMap, a novel open-vocabulary mapping approach that enhances instruction grounding through structural-semantic constraints and LLM-assisted instance selection.

Findings

Outperforms state-of-the-art in zero-shot instruction grounding

Effective in 3D semantic mapping and retrieval tasks

Demonstrates robustness across diverse indoor environments

Abstract

Grounding natural language instructions to visual observations is fundamental for embodied agents operating in open-world environments. Recent advances in visual-language mapping have enabled generalizable semantic representations by leveraging vision-language models (VLMs). However, these methods often fall short in aligning free-form language commands with specific scene instances, due to limitations in both instance-level semantic consistency and instruction interpretation. We present OpenMap, a zero-shot open-vocabulary visual-language map designed for accurate instruction grounding in navigation tasks. To address semantic inconsistencies across views, we introduce a Structural-Semantic Consensus constraint that jointly considers global geometric structure and vision-language similarity to guide robust 3D instance-level aggregation. To improve instruction interpretation, we propose an LLM-assisted Instruction-to-Instance Grounding module that enables fine-grained instance selection by incorporating spatial context and expressive target descriptions. We evaluate OpenMap on ScanNet200 and Matterport3D, covering both semantic mapping and instruction-to-target retrieval tasks. Experimental results show that OpenMap outperforms state-of-the-art baselines in zero-shot settings, demonstrating the effectiveness of our method in bridging free-form language and 3D perception for embodied navigation.