Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videos

TL;DR

This paper introduces a multimodal event knowledge graph and a hierarchical retrieval method to improve vision-language navigation, enabling agents to better handle long-horizon reasoning and ambiguous instructions in unseen indoor environments.

Contribution

It presents the first large-scale multimodal spatiotemporal knowledge graph and a novel retrieval mechanism that enhances VLN agents' reasoning capabilities.

Findings

Outperforms state-of-the-art methods on REVERIE, R2R, and R2R-CE benchmarks.

Effectively retrieves causal event sequences for improved navigation.

Demonstrates the benefit of multimodal episodic memory in complex reasoning tasks.

Abstract

Vision-Language Navigation (VLN) agents often struggle with long-horizon reasoning in unseen environments, particularly when facing ambiguous, coarse-grained instructions. While recent advances use knowledge graph to enhance reasoning, the potential of multimodal event knowledge inspired by human episodic memory remains underexplored. In this work, we propose an event-centric knowledge enhancement strategy for automated process knowledge mining and feature fusion to solve coarse-grained instruction and long-horizon reasoning in VLN task. First, we construct YE-KG, the first large-scale multimodal spatiotemporal knowledge graph, with over 86k nodes and 83k edges, derived from real-world indoor videos. By leveraging multimodal large language models (i.e., LLaVa, GPT4), we extract unstructured video streams into structured semantic-action-effect events to serve as explicit episodic memory. Second, we introduce STE-VLN, which integrates the above graph into VLN models via a Coarse-to-Fine Hierarchical Retrieval mechanism. This allows agents to retrieve causal event sequences and dynamically fuse them with egocentric visual observations. Experiments on REVERIE, R2R, and R2R-CE benchmarks demonstrate the efficiency of our event-centric strategy, outperforming state-of-the-art approaches across diverse action spaces. Our data and code are available on the project website https://sites.google.com/view/y-event-kg/.