Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration

TL;DR

This paper introduces ProbES, a prompt-based self-exploration method that leverages a large-scale cross-modal pretrained model to generate training data and adapt quickly to new environments in vision-language navigation tasks.

Contribution

The paper proposes a novel prompt-based self-exploration approach that eliminates the need for human-labeled data and enhances cross-domain adaptation in VLN tasks.

Findings

ProbES improves generalization in unseen environments.

It enables automatic environment exploration and instruction generation.

The method enhances adaptation speed and efficiency.

Abstract

Vision-language navigation (VLN) is a challenging task due to its large searching space in the environment. To address this problem, previous works have proposed some methods of fine-tuning a large model that pretrained on large-scale datasets. However, the conventional fine-tuning methods require extra human-labeled navigation data and lack self-exploration capabilities in environments, which hinders their generalization of unseen scenes. To improve the ability of fast cross-domain adaptation, we propose Prompt-based Environmental Self-exploration (ProbES), which can self-explore the environments by sampling trajectories and automatically generates structured instructions via a large-scale cross-modal pretrained model (CLIP). Our method fully utilizes the knowledge learned from CLIP to build an in-domain dataset by self-exploration without human labeling. Unlike the conventional approach of fine-tuning, we introduce prompt-based learning to achieve fast adaptation for language embeddings, which substantially improves the learning efficiency by leveraging prior knowledge. By automatically synthesizing trajectory-instruction pairs in any environment without human supervision and efficient prompt-based learning, our model can adapt to diverse vision-language navigation tasks, including VLN and REVERIE. Both qualitative and quantitative results show that our ProbES significantly improves the generalization ability of the navigation model.