ESceme: Vision-and-Language Navigation with Episodic Scene Memory

TL;DR

This paper introduces ESceme, a novel episodic scene memory mechanism for vision-and-language navigation that improves an agent's ability to utilize past visited scenes, enhancing navigation performance across multiple benchmarks.

Contribution

We propose ESceme, a simple yet effective episodic memory system that enhances VLN agents by enabling dynamic recall of past scenes, leading to better navigation accuracy.

Findings

ESceme outperforms existing methods on R2R, R4R, and CVDN datasets.

The approach achieves first place on the CVDN leaderboard.

Enhanced scene memory improves navigation in both short- and long-horizon tasks.

Abstract

Vision-and-language navigation (VLN) simulates a visual agent that follows natural-language navigation instructions in real-world scenes. Existing approaches have made enormous progress in navigation in new environments, such as beam search, pre-exploration, and dynamic or hierarchical history encoding. To balance generalization and efficiency, we resort to memorizing visited scenarios apart from the ongoing route while navigating. In this work, we introduce a mechanism of Episodic Scene memory (ESceme) for VLN that wakes an agent's memories of past visits when it enters the current scene. The episodic scene memory allows the agent to envision a bigger picture of the next prediction. This way, the agent learns to utilize dynamically updated information instead of merely adapting to the current observations. We provide a simple yet effective implementation of ESceme by enhancing the accessible views at each location and progressively completing the memory while navigating. We verify the superiority of ESceme on short-horizon (R2R), long-horizon (R4R), and vision-and-dialog (CVDN) VLN tasks. Our ESceme also wins first place on the CVDN leaderboard. Code is available: \url{https://github.com/qizhust/esceme}.