Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation

TL;DR

This paper introduces AO-Planner, a novel zero-shot affordances-oriented planning method for continuous vision-language navigation, integrating foundation models for low-level control and high-level decision-making, achieving state-of-the-art results.

Contribution

The paper presents AO-Planner, combining foundation models for low-level motion planning and high-level reasoning in continuous VLN, bridging the gap between high-level task planning and low-level control.

Findings

Achieves 8.8% improvement on SPL in R2R-CE and RxR-CE datasets.

Can serve as a data annotator for pseudo-label generation.

Attains 47% success rate with a data-efficient predictor.

Abstract

LLM-based agents have demonstrated impressive zero-shot performance in vision-language navigation (VLN) task. However, existing LLM-based methods often focus only on solving high-level task planning by selecting nodes in predefined navigation graphs for movements, overlooking low-level control in navigation scenarios. To bridge this gap, we propose AO-Planner, a novel Affordances-Oriented Planner for continuous VLN task. Our AO-Planner integrates various foundation models to achieve affordances-oriented low-level motion planning and high-level decision-making, both performed in a zero-shot setting. Specifically, we employ a Visual Affordances Prompting (VAP) approach, where the visible ground is segmented by SAM to provide navigational affordances, based on which the LLM selects potential candidate waypoints and plans low-level paths towards selected waypoints. We further propose a high-level PathAgent which marks planned paths into the image input and reasons the most probable path by comprehending all environmental information. Finally, we convert the selected path into 3D coordinates using camera intrinsic parameters and depth information, avoiding challenging 3D predictions for LLMs. Experiments on the challenging R2R-CE and RxR-CE datasets show that AO-Planner achieves state-of-the-art zero-shot performance (8.8% improvement on SPL). Our method can also serve as a data annotator to obtain pseudo-labels, distilling its waypoint prediction ability into a learning-based predictor. This new predictor does not require any waypoint data from the simulator and achieves 47% SR competing with supervised methods. We establish an effective connection between LLM and 3D world, presenting novel prospects for employing foundation models in low-level motion control.