A Navigation Framework Utilizing Vision-Language Models

TL;DR

This paper introduces a modular navigation framework that leverages large vision-language models to improve embodied AI navigation tasks, emphasizing efficiency and adaptability in complex environments.

Contribution

It proposes a decoupled, plug-and-play architecture integrating LVLMs with lightweight planning, enabling flexible navigation without extensive fine-tuning.

Findings

Achieved promising results on the Room-to-Room benchmark.

Identified challenges in generalizing to unseen environments.

Highlighted the potential of modular approaches for scalable navigation.

Abstract

Vision-and-Language Navigation (VLN) presents a complex challenge in embodied AI, requiring agents to interpret natural language instructions and navigate through visually rich, unfamiliar environments. Recent advances in large vision-language models (LVLMs), such as CLIP and Flamingo, have significantly improved multimodal understanding but introduced new challenges related to computational cost and real-time deployment. In this project, we propose a modular, plug-and-play navigation framework that decouples vision-language understanding from action planning. By integrating a frozen vision-language model, Qwen2.5-VL-7B-Instruct, with lightweight planning logic, we aim to achieve flexible, fast, and adaptable navigation without extensive model fine-tuning. Our framework leverages prompt engineering, structured history management, and a two-frame visual input strategy to enhance decision-making continuity across navigation steps. We evaluate our system on the Room-to-Room benchmark within the VLN-CE setting using the Matterport3D dataset and Habitat-Lab simulation environment. Although our initial results reveal challenges in generalizing to unseen environments under strict evaluation settings, our modular approach lays a foundation for scalable and efficient navigation systems, highlighting promising directions for future improvement through enhanced environmental priors and expanded multimodal input integration.