Navigating Beyond Instructions: Vision-and-Language Navigation in Obstructed Environments

TL;DR

This paper introduces R2R-UNO, a new dataset with obstructions for vision-and-language navigation, revealing current methods struggle with real-world obstacles, and proposes ObVLN, a training strategy to improve navigation in obstructed environments.

Contribution

The paper creates R2R-UNO dataset with diverse obstructions and proposes ObVLN, a novel training method, to enhance VLN agents' ability to navigate obstructed environments effectively.

Findings

State-of-the-art VLN methods struggle with obstructions.

ObVLN improves navigation performance in obstructed scenarios.

ObVLN maintains performance in unobstructed environments.

Abstract

Real-world navigation often involves dealing with unexpected obstructions such as closed doors, moved objects, and unpredictable entities. However, mainstream Vision-and-Language Navigation (VLN) tasks typically assume instructions perfectly align with the fixed and predefined navigation graphs without any obstructions. This assumption overlooks potential discrepancies in actual navigation graphs and given instructions, which can cause major failures for both indoor and outdoor agents. To address this issue, we integrate diverse obstructions into the R2R dataset by modifying both the navigation graphs and visual observations, introducing an innovative dataset and task, R2R with UNexpected Obstructions (R2R-UNO). R2R-UNO contains various types and numbers of path obstructions to generate instruction-reality mismatches for VLN research. Experiments on R2R-UNO reveal that state-of-the-art VLN methods inevitably encounter significant challenges when facing such mismatches, indicating that they rigidly follow instructions rather than navigate adaptively. Therefore, we propose a novel method called ObVLN (Obstructed VLN), which includes a curriculum training strategy and virtual graph construction to help agents effectively adapt to obstructed environments. Empirical results show that ObVLN not only maintains robust performance in unobstructed scenarios but also achieves a substantial performance advantage with unexpected obstructions.