HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System

TL;DR

HiMemVLN introduces a hierarchical memory system into open-source vision-and-language navigation models, significantly improving their ability to recall visual and spatial information, thereby closing the performance gap with closed-source models.

Contribution

This work presents HiMemVLN, a novel hierarchical memory system that enhances open-source VLN agents' visual perception and localization capabilities.

Findings

Nearly doubled performance over state-of-the-art open-source methods.

Effectively mitigates navigation amnesia and failure cases.

Demonstrates robustness in both simulated and real-world environments.

Abstract

LLM-based agents have demonstrated impressive zero-shot performance in vision-language navigation (VLN) tasks. However, most zero-shot methods primarily rely on closed-source LLMs as navigators, which face challenges related to high token costs and potential data leakage risks. Recent efforts have attempted to address this by using open-source LLMs combined with a spatiotemporal CoT framework, but they still fall far short compared to closed-source models. In this work, we identify a critical issue, Navigation Amnesia, through a detailed analysis of the navigation process. This issue leads to navigation failures and amplifies the gap between open-source and closed-source methods. To address this, we propose HiMemVLN, which incorporates a Hierarchical Memory System into a multimodal large model to enhance visual perception recall and long-term localization, mitigating the amnesia issue and improving the agent's navigation performance. Extensive experiments in both simulated and real-world environments demonstrate that HiMemVLN achieves nearly twice the performance of the open-source state-of-the-art method. The code is available at https://github.com/lvkailin0118/HiMemVLN.