DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigation

TL;DR

The paper introduces DOPE, a novel network that enhances vision-and-language navigation by better extracting instruction details and modeling object relationships across modalities, leading to improved navigation accuracy.

Contribution

The paper proposes a dual object perception-enhancement network with text and image modules to better utilize instruction details and object relationships in VLN tasks.

Findings

Improved navigation accuracy on R2R and REVERIE datasets.

Enhanced understanding of object relationships across modalities.

Effective extraction of essential instruction phrases.

Abstract

Vision-and-Language Navigation (VLN) is a challenging task where an agent must understand language instructions and navigate unfamiliar environments using visual cues. The agent must accurately locate the target based on visual information from the environment and complete tasks through interaction with the surroundings. Despite significant advancements in this field, two major limitations persist: (1) Many existing methods input complete language instructions directly into multi-layer Transformer networks without fully exploiting the detailed information within the instructions, thereby limiting the agent's language understanding capabilities during task execution; (2) Current approaches often overlook the modeling of object relationships across different modalities, failing to effectively utilize latent clues between objects, which affects the accuracy and robustness of navigation decisions. We propose a Dual Object Perception-Enhancement Network (DOPE) to address these issues to improve navigation performance. First, we design a Text Semantic Extraction (TSE) to extract relatively essential phrases from the text and input them into the Text Object Perception-Augmentation (TOPA) to fully leverage details such as objects and actions within the instructions. Second, we introduce an Image Object Perception-Augmentation (IOPA), which performs additional modeling of object information across different modalities, enabling the model to more effectively utilize latent clues between objects in images and text, enhancing decision-making accuracy. Extensive experiments on the R2R and REVERIE datasets validate the efficacy of the proposed approach.