osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning

TL;DR

This paper introduces osmAG-LLM, a novel robot navigation system that leverages semantic maps and large language models to effectively find objects in dynamic environments without relying on detailed maps.

Contribution

It presents a new approach that combines semantic mapping with LLM reasoning for open-vocabulary object navigation, addressing issues of map staleness and object movement.

Findings

Higher success in static object retrieval at shorter paths

Outperforms prior methods in dynamic or unmapped object scenarios

Effective in real-world and simulated environments

Abstract

Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features, achieving a high level of detail and guiding robots to find objects specified by open-vocabulary language queries. While the issue of scalability for such approaches has received some attention, another fundamental problem is that high-detail object mapping quickly becomes outdated, as objects get moved around a lot. In this work, we develop a mapping and navigation system for object-goal navigation that, from the ground up, considers the possibilities that a queried object can have moved, or may not be mapped at all. Instead of striving for high-fidelity mapping detail, we consider that the main purpose of a map is to provide environment grounding and context, which we combine with the semantic priors of LLMs to reason about object locations and deploy an active, online approach to navigate to the objects. Through simulated and real-world experiments we find that our approach tends to have higher retrieval success at shorter path lengths for static objects and by far outperforms prior approaches in cases of dynamic or unmapped object queries. We provide our code and dataset at: https://github.com/xiexiexiaoxiexie/osmAG-LLM.