CityNavAgent: Aerial Vision-and-Language Navigation with Hierarchical Semantic Planning and Global Memory

TL;DR

CityNavAgent introduces hierarchical semantic planning and global memory modules powered by large language models to improve aerial vision-and-language navigation in complex urban environments, achieving state-of-the-art results.

Contribution

The paper presents a novel hierarchical semantic planning module and a global memory system for aerial VLN, significantly reducing navigation complexity and enhancing performance.

Findings

Achieves state-of-the-art performance on aerial VLN benchmarks.

Hierarchical planning effectively decomposes long-horizon navigation tasks.

Global memory simplifies navigation by storing historical trajectories.

Abstract

Aerial vision-and-language navigation (VLN), requiring drones to interpret natural language instructions and navigate complex urban environments, emerges as a critical embodied AI challenge that bridges human-robot interaction, 3D spatial reasoning, and real-world deployment. Although existing ground VLN agents achieved notable results in indoor and outdoor settings, they struggle in aerial VLN due to the absence of predefined navigation graphs and the exponentially expanding action space in long-horizon exploration. In this work, we propose \textbf{CityNavAgent}, a large language model (LLM)-empowered agent that significantly reduces the navigation complexity for urban aerial VLN. Specifically, we design a hierarchical semantic planning module (HSPM) that decomposes the long-horizon task into sub-goals with different semantic levels. The agent reaches the target progressively by achieving sub-goals with different capacities of the LLM. Additionally, a global memory module storing historical trajectories into a topological graph is developed to simplify navigation for visited targets. Extensive benchmark experiments show that our method achieves state-of-the-art performance with significant improvement. Further experiments demonstrate the effectiveness of different modules of CityNavAgent for aerial VLN in continuous city environments. The code is available at \href{https://github.com/VinceOuti/CityNavAgent}{link}.