World-aware Planning Narratives Enhance Large Vision-Language Model Planner

TL;DR

This paper introduces WAP, a framework that enhances large vision-language models with environmental understanding, significantly improving their performance on complex embodied planning tasks involving visual reasoning and multi-step goals.

Contribution

The paper proposes a novel framework, WAP, that infuses LVLMs with environmental awareness through four cognitive capabilities, leading to substantial performance improvements on benchmarks.

Findings

Qwen2.5-VL achieves 60.7% success rate on EB-ALFRED.

Model performance improves notably in commonsense reasoning and long-horizon planning.

Enhanced models outperform proprietary systems like GPT-4o and Claude-3.5-Sonnet.

Abstract

Large Vision-Language Models (LVLMs) show promise for embodied planning tasks but struggle with complex scenarios involving unfamiliar environments and multi-step goals. Current approaches rely on environment-agnostic imitation learning that disconnects instructions from environmental contexts, causing models to struggle with context-sensitive instructions and rely on supplementary cues rather than visual reasoning during long-horizon interactions. In this work, we propose World-Aware Planning Narrative Enhancement (WAP), a framework that infuses LVLMs with comprehensive environmental understanding through four cognitive capabilities (visual appearance modeling, spatial reasoning, functional abstraction, and syntactic grounding) while developing and evaluating models using only raw visual observations through curriculum learning. Evaluations on the EB-ALFRED benchmark demonstrate substantial improvements, with Qwen2.5-VL achieving a 60.7 absolute improvement in task success rates, particularly in commonsense reasoning (+60.0) and long-horizon planning (+70.0). Notably, our enhanced open-source models outperform proprietary systems like GPT-4o and Claude-3.5-Sonnet by a large margin.