Recursive Visual Imagination and Adaptive Linguistic Grounding for Vision Language Navigation

TL;DR

This paper introduces a recursive visual imagination and adaptive linguistic grounding approach to improve vision language navigation by better organizing visual observations and aligning them with commands, leading to more accurate navigation in complex scenes.

Contribution

The paper proposes a novel recursive visual imagination method and adaptive linguistic grounding technique to enhance scene understanding and command alignment in VLN tasks.

Findings

Outperforms state-of-the-art on VLN-CE and ObjectNav benchmarks.

Improves scene representation by focusing on semantic layouts.

Enhances command comprehension through fine-grained semantic matching.

Abstract

Vision Language Navigation (VLN) typically requires agents to navigate to specified objects or remote regions in unknown scenes by obeying linguistic commands. Such tasks require organizing historical visual observations for linguistic grounding, which is critical for long-sequence navigational decisions. However, current agents suffer from overly detailed scene representation and ambiguous vision-language alignment, which weaken their comprehension of navigation-friendly high-level scene priors and easily lead to behaviors that violate linguistic commands. To tackle these issues, we propose a navigation policy by recursively summarizing along-the-way visual perceptions, which are adaptively aligned with commands to enhance linguistic grounding. In particular, by structurally modeling historical trajectories as compact neural grids, several Recursive Visual Imagination (RVI) techniques are proposed to motivate agents to focus on the regularity of visual transitions and semantic scene layouts, instead of dealing with misleading geometric details. Then, an Adaptive Linguistic Grounding (ALG) technique is proposed to align the learned situational memories with different linguistic components purposefully. Such fine-grained semantic matching facilitates the accurate anticipation of navigation actions and progress. Our navigation policy outperforms the state-of-the-art methods on the challenging VLN-CE and ObjectNav tasks, showing the superiority of our RVI and ALG techniques for VLN.