ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

TL;DR

ViPlan introduces a new benchmark to compare symbolic and direct vision-language model planning methods across visual domains, revealing strengths and limitations of each approach in different scenarios.

Contribution

This work provides the first open-source benchmark for evaluating VLM-grounded symbolic planning versus direct VLM planning in visual tasks.

Findings

VLM-as-grounder outperforms in Blocksworld (46% vs 9%)

VLM-as-planner outperforms in household robotics (34% vs 5%)

Chain-of-Thought prompting shows no consistent benefit

Abstract

Integrating Large Language Models with symbolic planners is a promising direction for obtaining verifiable and grounded plans, with recent work extending this idea to visual domains using Vision-Language Models (VLMs). However, a rigorous comparison with methods that plan directly with VLMs is missing, due to a lack of visual benchmarks that support symbolic planning. We present ViPlan, the first open-source benchmark for comparing VLM-grounded symbolic approaches (VLM-as-grounder) with direct VLM planning methods (VLM-as-planner). ViPlan introduces a series of increasingly challenging tasks in two visual domains: a visual variant of the classic Blocksworld planning problem and a simulated household robotics environment. We find VLM-as-grounder methods to outperform direct VLM planning in Blocksworld (solving 46% of the tasks against 9%), where image grounding is both crucial and accurate. However, in the household robotics tasks, where linguistic knowledge helps, VLM-as-planner methods are greatly superior to VLM-as-grounder approaches (solving 34% of the tasks against 5%), which are hindered by partial observability. Thus, ViPlan domains capture fundamental shortcomings of both planning approaches, which we further diagnose with a qualitative failure analysis. Finally, across methods, we observe no consistent benefit from Chain-of-Thought prompting, suggesting persistent limitations in current VLMs' visual reasoning abilities.