Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation

TL;DR

This paper introduces R3, a dual-process thinking system combining lightweight models and large language models for improved vision-and-language navigation, achieving significant performance gains on benchmark tasks.

Contribution

The paper proposes a novel dual-process framework R3 that integrates a lightweight expert model with a large language model for zero-shot VLN, addressing efficiency and comprehension issues.

Findings

R3 outperforms state-of-the-art methods on REVERIE benchmark.

The framework achieves over 3.28% improvement in SPL.

The approach effectively balances accuracy and computational efficiency.

Abstract

Vision-and-Language Navigation (VLN) requires an agent to dynamically explore complex 3D environments following human instructions. Recent research underscores the potential of harnessing large language models (LLMs) for VLN, given their commonsense knowledge and general reasoning capabilities. Despite their strengths, a substantial gap in task completion performance persists between LLM-based approaches and domain experts, as LLMs inherently struggle to comprehend real-world spatial correlations precisely. Additionally, introducing LLMs is accompanied with substantial computational cost and inference latency. To address these issues, we propose a novel dual-process thinking framework dubbed R3, integrating LLMs' generalization capabilities with VLN-specific expertise in a zero-shot manner. The framework comprises three core modules: Runner, Ruminator, and Regulator. The Runner is a lightweight transformer-based expert model that ensures efficient and accurate navigation under regular circumstances. The Ruminator employs a powerful multimodal LLM as the backbone and adopts chain-of-thought (CoT) prompting to elicit structured reasoning. The Regulator monitors the navigation progress and controls the appropriate thinking mode according to three criteria, integrating Runner and Ruminator harmoniously. Experimental results illustrate that R3 significantly outperforms other state-of-the-art methods, exceeding 3.28% and 3.30% in SPL and RGSPL respectively on the REVERIE benchmark. This pronounced enhancement highlights the effectiveness of our method in handling challenging VLN tasks.