NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM

TL;DR

NavRAG introduces a retrieval-augmented generation framework that creates diverse, user-demand-oriented instructions for embodied navigation, enhancing data quality and model performance in vision-and-language navigation tasks.

Contribution

The paper presents NavRAG, a novel RAG framework that generates high-quality, diverse navigation instructions by leveraging LLMs and scene retrieval, addressing limitations of previous data augmentation methods.

Findings

Annotated over 2 million navigation instructions across 861 scenes.

NavRAG improves navigation model performance with diverse instruction data.

Generated instructions better match user communication styles.

Abstract

Vision-and-Language Navigation (VLN) is an essential skill for embodied agents, allowing them to navigate in 3D environments following natural language instructions. High-performance navigation models require a large amount of training data, the high cost of manually annotating data has seriously hindered this field. Therefore, some previous methods translate trajectory videos into step-by-step instructions for expanding data, but such instructions do not match well with users' communication styles that briefly describe destinations or state specific needs. Moreover, local navigation trajectories overlook global context and high-level task planning. To address these issues, we propose NavRAG, a retrieval-augmented generation (RAG) framework that generates user demand instructions for VLN. NavRAG leverages LLM to build a hierarchical scene description tree for 3D scene understanding from global layout to local details, then simulates various user roles with specific demands to retrieve from the scene tree, generating diverse instructions with LLM. We annotate over 2 million navigation instructions across 861 scenes and evaluate the data quality and navigation performance of trained models.