FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation

TL;DR

FOAM introduces a follower-aware speaker model that dynamically updates based on follower feedback, leading to more effective instruction generation and improved navigation performance in vision-and-language tasks.

Contribution

The paper proposes a bi-level optimization framework for a follower-aware speaker model that adapts during training to produce better instructions for navigation.

Findings

Outperforms baseline models on Room-to-Room and Room-across-Room datasets.

Generated instructions are of higher quality than previous methods.

Enhances navigation accuracy by integrating follower feedback into instruction synthesis.

Abstract

The speaker-follower models have proven to be effective in vision-and-language navigation, where a speaker model is used to synthesize new instructions to augment the training data for a follower navigation model. However, in many of the previous methods, the generated instructions are not directly trained to optimize the performance of the follower. In this paper, we present \textsc{foam}, a \textsc{Fo}llower-\textsc{a}ware speaker \textsc{M}odel that is constantly updated given the follower feedback, so that the generated instructions can be more suitable to the current learning state of the follower. Specifically, we optimize the speaker using a bi-level optimization framework and obtain its training signals by evaluating the follower on labeled data. Experimental results on the Room-to-Room and Room-across-Room datasets demonstrate that our methods can outperform strong baseline models across settings. Analyses also reveal that our generated instructions are of higher quality than the baselines.