LOViS: Learning Orientation and Visual Signals for Vision and Language Navigation

TL;DR

This paper introduces LOViS, a neural navigation agent with separate orientation and vision modules, improving spatial reasoning and visual perception for better performance in vision-and-language navigation tasks.

Contribution

The paper proposes a novel neural architecture with explicit orientation and vision modules, along with specialized pre-training tasks, to enhance spatial grounding and visual understanding in navigation.

Findings

Achieves state-of-the-art results on R2R and R4R datasets.

Demonstrates improved spatial reasoning and landmark grounding.

Enhances the learning of orientation and visual signals separately.

Abstract

Understanding spatial and visual information is essential for a navigation agent who follows natural language instructions. The current Transformer-based VLN agents entangle the orientation and vision information, which limits the gain from the learning of each information source. In this paper, we design a neural agent with explicit Orientation and Vision modules. Those modules learn to ground spatial information and landmark mentions in the instructions to the visual environment more effectively. To strengthen the spatial reasoning and visual perception of the agent, we design specific pre-training tasks to feed and better utilize the corresponding modules in our final navigation model. We evaluate our approach on both Room2room (R2R) and Room4room (R4R) datasets and achieve the state of the art results on both benchmarks.