Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotations

TL;DR

This paper introduces FCA-NIG, a generative framework that automatically creates fine-grained, annotated vision-language navigation instructions, significantly improving agent performance by providing detailed sub-instruction and entity-landmark alignments.

Contribution

The paper presents FCA-NIG, the first large-scale dataset augmentation method for fine-grained cross-modal alignments in VLN, enhancing training data quality without manual annotation.

Findings

Training with FCA-R2R improves VLN agent performance.

Sub-instruction alignment increases state awareness and decision accuracy.

Entity-landmark alignment enhances navigation performance and generalization.

Abstract

Vision-Language Navigation (VLN) enables intelligent agents to navigate environments by integrating visual perception and natural language instructions, yet faces significant challenges due to the scarcity of fine-grained cross-modal alignment annotations. Existing datasets primarily focus on global instruction-trajectory matching, neglecting sub-instruction-level and entity-level alignments critical for accurate navigation action decision-making. To address this limitation, we propose FCA-NIG, a generative framework that automatically constructs navigation instructions with dual-level fine-grained cross-modal annotations. In this framework, an augmented trajectory is first divided into sub-trajectories, which are then processed through GLIP-based landmark detection, crafted instruction construction, OFA-Speaker based R2R-like instruction generation, and CLIP-powered entity selection, generating sub-instruction-trajectory pairs with entity-landmark annotations. Finally, these sub-pairs are aggregated to form a complete instruction-trajectory pair. The framework generates the FCA-R2R dataset, the first large-scale augmentation dataset featuring precise sub-instruction-sub-trajectory and entity-landmark alignments. Extensive experiments demonstrate that training with FCA-R2R significantly improves the performance of multiple state-of-the-art VLN agents, including SF, EnvDrop, RecBERT, and HAMT. Incorporating sub-instruction-trajectory alignment enhances agents' state awareness and decision accuracy, while entity-landmark alignment further boosts navigation performance and generalization. These results highlight the effectiveness of FCA-NIG in generating high-quality, scalable training data without manual annotation, advancing fine-grained cross-modal learning in complex navigation tasks.