Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation

TL;DR

This paper introduces MTVM, a multimodal transformer with variable-length memory for vision-and-language navigation, explicitly modeling long-term temporal context to improve navigation performance.

Contribution

It proposes a novel MTVM model that stores navigation history in a memory bank and uses a memory-aware loss, enhancing temporal context modeling in VLN tasks.

Findings

Improves Success Rate by 2% on R2R unseen validation and test sets.

Reduces Goal Process by 1.6 million steps on CVDN test set.

Outperforms previous methods in long-term temporal context understanding.

Abstract

Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position, which relies on the ongoing interactions with the environment during moving. Recent Transformer-based VLN methods have made great progress benefiting from the direct connections between visual observations and the language instruction via the multimodal cross-attention mechanism. However, these methods usually represent temporal context as a fixed-length vector by using an LSTM decoder or using manually designed hidden states to build a recurrent Transformer. Considering a single fixed-length vector is often insufficient to capture long-term temporal context, in this paper, we introduce Multimodal Transformer with Variable-length Memory (MTVM) for visually-grounded natural language navigation by modelling the temporal context explicitly. Specifically, MTVM enables the agent to keep track of the navigation trajectory by directly storing previous activations in a memory bank. To further boost the performance, we propose a memory-aware consistency loss to help learn a better joint representation of temporal context with random masked instructions. We evaluate MTVM on popular R2R and CVDN datasets, and our model improves Success Rate on R2R unseen validation and test set by 2% each, and reduce Goal Process by 1.6m on CVDN test set.