Towards Coarse-grained Visual Language Navigation Task Planning Enhanced by Event Knowledge Graph

TL;DR

This paper introduces a novel approach for visual language navigation that leverages an event knowledge graph to improve planning with coarse-grained instructions, demonstrating significant success rate improvements.

Contribution

It proposes a prompt-based framework to extract an event knowledge graph and integrates it into VLN for better handling of abstract, coarse-grained instructions.

Findings

Over 5% improvement in success rate on benchmark datasets.

Effective knowledge-enhanced navigation planning with event knowledge graph.

Real-time error correction via dynamic history backtracking.

Abstract

Visual language navigation (VLN) is one of the important research in embodied AI. It aims to enable an agent to understand the surrounding environment and complete navigation tasks. VLN instructions could be categorized into coarse-grained and fine-grained commands. Fine-grained command describes a whole task with subtasks step-by-step. In contrast, coarse-grained command gives an abstract task description, which more suites human habits. Most existing work focuses on the former kind of instruction in VLN tasks, ignoring the latter abstract instructions belonging to daily life scenarios. To overcome the above challenge in abstract instruction, we attempt to consider coarse-grained instruction in VLN by event knowledge enhancement. Specifically, we first propose a prompt-based framework to extract an event knowledge graph (named VLN-EventKG) for VLN integrally over multiple mainstream benchmark datasets. Through small and large language model collaboration, we realize knowledge-enhanced navigation planning (named EventNav) for VLN tasks with coarse-grained instruction input. Additionally, we design a novel dynamic history backtracking module to correct potential error action planning in real time. Experimental results in various public benchmarks show our knowledge-enhanced method has superiority in coarse-grained-instruction VLN using our proposed VLN-EventKG with over $5\%$ improvement in success rate. Our project is available at https://sites.google.com/view/vln-eventkg