Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments

TL;DR

This paper introduces Ground-level Viewpoint Navigation (GVNav), a novel approach for vision-and-language navigation tailored to low-height robots, addressing generalization challenges across diverse environments and viewpoints.

Contribution

It proposes weighted historical observations and graph transfer techniques to improve low-height robot navigation and generalization in realistic settings.

Findings

Enhanced navigation accuracy in simulated environments.

Improved real-world deployment performance.

Effective handling of visual obstructions and perceptual mismatches.

Abstract

Vision-and-Language Navigation (VLN) empowers agents to associate time-sequenced visual observations with corresponding instructions to make sequential decisions. However, generalization remains a persistent challenge, particularly when dealing with visually diverse scenes or transitioning from simulated environments to real-world deployment. In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view, proposing a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue. This work represents the first attempt to highlight the generalization gap in VLN across varying heights of visual observation in realistic robot deployments. Our approach leverages weighted historical observations as enriched spatiotemporal contexts for instruction following, effectively managing feature collisions within cells by assigning appropriate weights to identical features across different viewpoints. This enables low-height robots to overcome challenges such as visual obstructions and perceptual mismatches. Additionally, we transfer the connectivity graph from the HM3D and Gibson datasets as an extra resource to enhance spatial priors and a more comprehensive representation of real-world scenarios, leading to improved performance and generalizability of the waypoint predictor in real-world environments. Extensive experiments demonstrate that our Ground-level Viewpoint Navigation (GVnav) approach significantly improves performance in both simulated environments and real-world deployments with quadruped robots.