Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation

TL;DR

This paper introduces a novel pre-training paradigm called GELA for Vision-and-Language Navigation, focusing on fine-grained entity-landmark alignment to improve navigation accuracy.

Contribution

It proposes grounded entity-landmark annotations and adaptive pre-training objectives, enhancing cross-modal alignment at the entity level in VLN tasks.

Findings

Achieves state-of-the-art results on R2R and CVDN benchmarks.

Demonstrates improved fine-grained entity-landmark alignment.

Validates effectiveness and generalizability of GELA approach.

Abstract

Cross-modal alignment is one key challenge for Vision-and-Language Navigation (VLN). Most existing studies concentrate on mapping the global instruction or single sub-instruction to the corresponding trajectory. However, another critical problem of achieving fine-grained alignment at the entity level is seldom considered. To address this problem, we propose a novel Grounded Entity-Landmark Adaptive (GELA) pre-training paradigm for VLN tasks. To achieve the adaptive pre-training paradigm, we first introduce grounded entity-landmark human annotations into the Room-to-Room (R2R) dataset, named GEL-R2R. Additionally, we adopt three grounded entity-landmark adaptive pre-training objectives: 1) entity phrase prediction, 2) landmark bounding box prediction, and 3) entity-landmark semantic alignment, which explicitly supervise the learning of fine-grained cross-modal alignment between entity phrases and environment landmarks. Finally, we validate our model on two downstream benchmarks: VLN with descriptive instructions (R2R) and dialogue instructions (CVDN). The comprehensive experiments show that our GELA model achieves state-of-the-art results on both tasks, demonstrating its effectiveness and generalizability.