VPN: Visual Prompt Navigation

TL;DR

VPN introduces a visual prompt-based navigation paradigm that uses intuitive visual cues on top-view maps, reducing language ambiguity and improving navigation in complex environments, with new datasets and a specialized baseline network.

Contribution

The paper proposes a novel visual prompt navigation paradigm, creates new datasets, and develops a baseline network with data augmentation strategies for improved navigation performance.

Findings

Visual prompts improve navigation accuracy over language instructions.

Data augmentation strategies enhance the robustness of VPN models.

VPN outperforms baseline methods in new datasets.

Abstract

While natural language is commonly used to guide embodied agents, the inherent ambiguity and verbosity of language often hinder the effectiveness of language-guided navigation in complex environments. To this end, we propose Visual Prompt Navigation (VPN), a novel paradigm that guides agents to navigate using only user-provided visual prompts within 2D top-view maps. This visual prompt primarily focuses on marking the visual navigation trajectory on a top-down view of a scene, offering intuitive and spatially grounded guidance without relying on language instructions. It is more friendly for non-expert users and reduces interpretive ambiguity. We build VPN tasks in both discrete and continuous navigation settings, constructing two new datasets, R2R-VP and R2R-CE-VP, by extending existing R2R and R2R-CE episodes with corresponding visual prompts. Furthermore, we introduce VPNet, a dedicated baseline network to handle the VPN tasks, with two data augmentation strategies: view-level augmentation (altering initial headings and prompt orientations) and trajectory-level augmentation (incorporating diverse trajectories from large-scale 3D scenes), to enhance navigation performance. Extensive experiments evaluate how visual prompt forms, top-view map formats, and data augmentation strategies affect the performance of visual prompt navigation. The code is available at https://github.com/farlit/VPN.