ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination

TL;DR

ImagineNav leverages vision-language models' spatial and planning abilities by enabling them to imagine future views, transforming navigation into a view selection task, and demonstrating superior performance in object navigation benchmarks.

Contribution

This work introduces ImagineNav, a novel framework that enhances VLMs with scene imagination for mapless visual navigation using only onboard RGB/RGB-D inputs.

Findings

Outperforms existing methods on open-vocabulary object navigation benchmarks.

Effectively transforms navigation planning into a view selection problem.

Demonstrates the potential of VLMs for embodied navigation tasks.

Abstract

Visual navigation is an essential skill for home-assistance robots, providing the object-searching ability to accomplish long-horizon daily tasks. Many recent approaches use Large Language Models (LLMs) for commonsense inference to improve exploration efficiency. However, the planning process of LLMs is limited within texts and it is difficult to represent the spatial occupancy and geometry layout only by texts. Both are important for making rational navigation decisions. In this work, we seek to unleash the spatial perception and planning ability of Vision-Language Models (VLMs), and explore whether the VLM, with only on-board camera captured RGB/RGB-D stream inputs, can efficiently finish the visual navigation tasks in a mapless manner. We achieve this by developing the imagination-powered navigation framework ImagineNav, which imagines the future observation images at valuable robot views and translates the complex navigation planning process into a rather simple best-view image selection problem for VLM. To generate appropriate candidate robot views for imagination, we introduce the Where2Imagine module, which is distilled to align with human navigation habits. Finally, to reach the VLM preferred views, an off-the-shelf point-goal navigation policy is utilized. Empirical experiments on the challenging open-vocabulary object navigation benchmarks demonstrates the superiority of our proposed system.