Fine-Grained Alignment in Vision-and-Language Navigation through Bayesian Optimization

TL;DR

This paper introduces a Bayesian Optimization-based adversarial framework to improve fine-grained alignment in vision-and-language navigation, enhancing cross-modal embeddings and navigation performance in 3D environments.

Contribution

It proposes a novel Bayesian Optimization approach for creating fine-grained contrastive vision samples to improve VLN embeddings.

Findings

Enhanced embeddings improve navigation accuracy.

Significant performance gains on R2R and REVERIE benchmarks.

Framework effectively generates fine-grained vision negatives.

Abstract

This paper addresses the challenge of fine-grained alignment in Vision-and-Language Navigation (VLN) tasks, where robots navigate realistic 3D environments based on natural language instructions. Current approaches use contrastive learning to align language with visual trajectory sequences. Nevertheless, they encounter difficulties with fine-grained vision negatives. To enhance cross-modal embeddings, we introduce a novel Bayesian Optimization-based adversarial optimization framework for creating fine-grained contrastive vision samples. To validate the proposed methodology, we conduct a series of experiments to assess the effectiveness of the enriched embeddings on fine-grained vision negatives. We conduct experiments on two common VLN benchmarks R2R and REVERIE, experiments on the them demonstrate that these embeddings benefit navigation, and can lead to a promising performance enhancement. Our source code and trained models are available at: https://anonymous.4open.science/r/FGVLN.