Beyond Visual Understanding: Introducing PARROT-360V for Vision Language Model Benchmarking

TL;DR

The paper introduces PARROT-360V, a comprehensive benchmark with 2487 visual puzzles designed to evaluate vision-language models on complex reasoning tasks, revealing significant performance gaps in current models.

Contribution

It presents a new challenging benchmark for VLMs that emphasizes complex visual reasoning and multi-modal integration, addressing limitations of existing evaluation methods.

Findings

State-of-the-art models scored 28-56%, lower than on standard benchmarks.

Current VLMs struggle with multi-step reasoning tasks.

The benchmark exposes the need for more robust evaluation frameworks.

Abstract

Current benchmarks for evaluating Vision Language Models (VLMs) often fall short in thoroughly assessing model abilities to understand and process complex visual and textual content. They typically focus on simple tasks that do not require deep reasoning or the integration of multiple data modalities to solve an original problem. To address this gap, we introduce the PARROT-360V Benchmark, a novel and comprehensive benchmark featuring 2487 challenging visual puzzles designed to test VLMs on complex visual reasoning tasks. We evaluated leading models: GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro, using PARROT-360V to assess their capabilities in combining visual clues with language skills to solve tasks in a manner akin to human problem-solving. Our findings reveal a notable performance gap: state-of-the-art models scored between 28 to 56 percentage on our benchmark, significantly lower than their performance on popular benchmarks. This underscores the limitations of current VLMs in handling complex, multi-step reasoning tasks and highlights the need for more robust evaluation frameworks to advance the field.