Cog-GA: A Large Language Models-based Generative Agent for Vision-Language Navigation in Continuous Environments

TL;DR

Cog-GA introduces a large language model-based generative agent for vision-language navigation in continuous 3D environments, employing cognitive mapping, waypoint prediction, and reflective mechanisms to enhance navigation efficiency and interpretability.

Contribution

This paper presents Cog-GA, a novel LLM-based agent that constructs cognitive maps and predicts waypoints, advancing the capabilities of VLN-CE agents with human-like reasoning.

Findings

Achieves state-of-the-art performance on VLN-CE benchmarks.

Demonstrates effective spatial reasoning and navigation efficiency.

Shows improved interpretability through dual-channel scene descriptions.

Abstract

Vision Language Navigation in Continuous Environments (VLN-CE) represents a frontier in embodied AI, demanding agents to navigate freely in unbounded 3D spaces solely guided by natural language instructions. This task introduces distinct challenges in multimodal comprehension, spatial reasoning, and decision-making. To address these challenges, we introduce Cog-GA, a generative agent founded on large language models (LLMs) tailored for VLN-CE tasks. Cog-GA employs a dual-pronged strategy to emulate human-like cognitive processes. Firstly, it constructs a cognitive map, integrating temporal, spatial, and semantic elements, thereby facilitating the development of spatial memory within LLMs. Secondly, Cog-GA employs a predictive mechanism for waypoints, strategically optimizing the exploration trajectory to maximize navigational efficiency. Each waypoint is accompanied by a dual-channel scene description, categorizing environmental cues into 'what' and 'where' streams as the brain. This segregation enhances the agent's attentional focus, enabling it to discern pertinent spatial information for navigation. A reflective mechanism complements these strategies by capturing feedback from prior navigation experiences, facilitating continual learning and adaptive replanning. Extensive evaluations conducted on VLN-CE benchmarks validate Cog-GA's state-of-the-art performance and ability to simulate human-like navigation behaviors. This research significantly contributes to the development of strategic and interpretable VLN-CE agents.