Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation

TL;DR

This paper introduces structure-encoding auxiliary tasks to pre-train image encoders using navigation environment data, significantly enhancing visual representations for improved performance in Vision-and-Language Navigation tasks.

Contribution

It proposes novel auxiliary tasks for pre-training image encoders with environment data, addressing the distribution shift issue in VLN and improving navigation success rates.

Findings

SEA pre-trained features encode scene structure better.

Improved success rates on Test-Unseen environments.

Plug-and-play with existing VLN agents without tuning.

Abstract

In Vision-and-Language Navigation (VLN), researchers typically take an image encoder pre-trained on ImageNet without fine-tuning on the environments that the agent will be trained or tested on. However, the distribution shift between the training images from ImageNet and the views in the navigation environments may render the ImageNet pre-trained image encoder suboptimal. Therefore, in this paper, we design a set of structure-encoding auxiliary tasks (SEA) that leverage the data in the navigation environments to pre-train and improve the image encoder. Specifically, we design and customize (1) 3D jigsaw, (2) traversability prediction, and (3) instance classification to pre-train the image encoder. Through rigorous ablations, our SEA pre-trained features are shown to better encode structural information of the scenes, which ImageNet pre-trained features fail to properly encode but is crucial for the target navigation task. The SEA pre-trained features can be easily plugged into existing VLN agents without any tuning. For example, on Test-Unseen environments, the VLN agents combined with our SEA pre-trained features achieve absolute success rate improvement of 12% for Speaker-Follower, 5% for Env-Dropout, and 4% for AuxRN.