Evaluating Vision-Language Models as Evaluators in Path Planning

TL;DR

This paper introduces PathEval, a benchmark to assess vision-language models as path evaluators in complex planning, revealing their current limitations and the need for task-specific vision adaptation.

Contribution

The work presents PathEval, a new benchmark for evaluating VLMs in path planning, and analyzes their performance, highlighting the importance of vision encoder adaptation.

Findings

VLMs struggle with low-level perception in path evaluation

Abstracting traits of optimal paths is feasible for VLMs

End-to-end fine-tuning is insufficient for improving VLMs in this task

Abstract

Despite their promise to perform complex reasoning, large language models (LLMs) have been shown to have limited effectiveness in end-to-end planning. This has inspired an intriguing question: if these models cannot plan well, can they still contribute to the planning framework as a helpful plan evaluator? In this work, we generalize this question to consider LLMs augmented with visual understanding, i.e., Vision-Language Models (VLMs). We introduce PathEval, a novel benchmark evaluating VLMs as plan evaluators in complex path-planning scenarios. Succeeding in the benchmark requires a VLM to be able to abstract traits of optimal paths from the scenario description, demonstrate precise low-level perception on each path, and integrate this information to decide the better path. Our analysis of state-of-the-art VLMs reveals that these models face significant challenges on the benchmark. We observe that the VLMs can precisely abstract given scenarios to identify the desired traits and exhibit mixed performance in integrating the provided information. Yet, their vision component presents a critical bottleneck, with models struggling to perceive low-level details about a path. Our experimental results show that this issue cannot be trivially addressed via end-to-end fine-tuning; rather, task-specific discriminative adaptation of these vision encoders is needed for these VLMs to become effective path evaluators.