Bird's-Eye-View Scene Graph for Vision-Language Navigation

TL;DR

This paper introduces a Bird's-Eye-View (BEV) scene graph for vision-language navigation, enabling better scene understanding and navigation accuracy by leveraging multi-step BEV representations and scene geometry.

Contribution

The paper proposes a novel BEV scene graph approach that encodes scene layouts and geometry, improving navigation performance over existing panoramic view-based methods.

Findings

Outperforms state-of-the-art on REVERIE, R2R, and R4R datasets.

Effectively encodes scene geometry and layout for navigation.

Enhances decision-making with BEV-based global scene maps.

Abstract

Vision-language navigation (VLN), which entails an agent to navigate 3D environments following human instructions, has shown great advances. However, current agents are built upon panoramic observations, which hinders their ability to perceive 3D scene geometry and easily leads to ambiguous selection of panoramic view. To address these limitations, we present a BEV Scene Graph (BSG), which leverages multi-step BEV representations to encode scene layouts and geometric cues of indoor environment under the supervision of 3D detection. During navigation, BSG builds a local BEV representation at each step and maintains a BEV-based global scene map, which stores and organizes all the online collected local BEV representations according to their topological relations. Based on BSG, the agent predicts a local BEV grid-level decision score and a global graph-level decision score, combined with a sub-view selection score on panoramic views, for more accurate action prediction. Our approach significantly outperforms state-of-the-art methods on REVERIE, R2R, and R4R, showing the potential of BEV perception in VLN.