Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models

TL;DR

The paper introduces the Visual Graph Arena (VGA), a dataset and framework for evaluating and improving AI models' ability to recognize and reason about visual concepts invariant to visual form variations, highlighting current limitations.

Contribution

It presents the VGA dataset with graph-based tasks to assess visual abstraction in AI, revealing significant gaps in current models' reasoning capabilities compared to humans.

Findings

Humans achieved near-perfect accuracy on VGA tasks.

State-of-the-art models failed on isomorphism detection.

Models showed limited success on path and cycle reasoning.

Abstract

Recent advancements in multimodal large language models have driven breakthroughs in visual question answering. Yet, a critical gap persists, `conceptualization'-the ability to recognize and reason about the same concept despite variations in visual form, a basic ability of human reasoning. To address this challenge, we introduce the Visual Graph Arena (VGA), a dataset featuring six graph-based tasks designed to evaluate and improve AI systems' capacity for visual abstraction. VGA uses diverse graph layouts (e.g., Kamada-Kawai vs. planar) to test reasoning independent of visual form. Experiments with state-of-the-art vision models and multimodal LLMs reveal a striking divide: humans achieved near-perfect accuracy across tasks, while models totally failed on isomorphism detection and showed limited success in path/cycle tasks. We further identify behavioral anomalies suggesting pseudo-intelligent pattern matching rather than genuine understanding. These findings underscore fundamental limitations in current AI models for visual understanding. By isolating the challenge of representation-invariant reasoning, the VGA provides a framework to drive progress toward human-like conceptualization in AI visual models. The Visual Graph Arena is available at: \href{https://vga.csail.mit.edu/}{vga.csail.mit.edu}