Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning

TL;DR

This paper benchmarks multimodal large language models for robotic path planning, revealing current limitations in scalability, spatial reasoning, and the utility of visual inputs across different grid complexities.

Contribution

It provides a comprehensive evaluation of 15 multimodal LLMs on robotic path planning tasks, highlighting challenges and potential benefits of visual inputs in this domain.

Findings

Visual inputs offer some benefits on small grids.

Performance degrades on larger, more complex grids.

Larger models perform better but are not always superior with visual data.

Abstract

Large Language Models (LLMs) show potential for enhancing robotic path planning. This paper assesses visual input's utility for multimodal LLMs in such tasks via a comprehensive benchmark. We evaluated 15 multimodal LLMs on generating valid and optimal paths in 2D grid environments, simulating simplified robotic planning, comparing text-only versus text-plus-visual inputs across varying model sizes and grid complexities. Our results indicate moderate success rates on simpler small grids, where visual input or few-shot text prompting offered some benefits. However, performance significantly degraded on larger grids, highlighting a scalability challenge. While larger models generally achieved higher average success, the visual modality was not universally dominant over well-structured text for these multimodal systems, and successful paths on simpler grids were generally of high quality. These results indicate current limitations in robust spatial reasoning, constraint adherence, and scalable multimodal integration, identifying areas for future LLM development in robotic path planning.