PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns

TL;DR

PuzzleVQA introduces a challenging dataset of abstract pattern puzzles to evaluate and diagnose the reasoning and perception capabilities of large multimodal models, revealing significant limitations in their generalization and reasoning skills.

Contribution

The paper presents PuzzleVQA, a novel dataset of 2000 abstract pattern puzzles designed to evaluate and analyze the reasoning abilities of large multimodal models.

Findings

State-of-the-art models struggle with abstract pattern generalization.

GPT-4V scores only 46.4% on single-concept puzzles.

Main bottlenecks are visual perception and inductive reasoning abilities.

Abstract

Large multimodal models extend the impressive capabilities of large language models by integrating multimodal understanding abilities. However, it is not clear how they can emulate the general intelligence and reasoning ability of humans. As recognizing patterns and abstracting concepts are key to general intelligence, we introduce PuzzleVQA, a collection of 2000 puzzle instances based on abstract patterns. With this dataset, we evaluate large multimodal models with abstract patterns based on fundamental concepts, including colors, numbers, sizes, and shapes. Through our experiments on state-of-the-art large multimodal models, we find that they are not able to generalize well to simple abstract patterns. Notably, GPT-4V achieves a score of 46.4% on single-concept puzzles, which shows that state-of-the-art models struggle on our dataset. To diagnose the reasoning challenges in large multimodal models, we progressively guide the models with our ground truth reasoning explanations for visual perception, inductive reasoning, and deductive reasoning. Our systematic analysis finds that the main bottlenecks of GPT-4V are weaker visual perception and inductive reasoning abilities. Through this work, we hope to shed light on the limitations of large multimodal models and how they can better emulate human cognitive processes in the future. Our data and code are available at https://puzzlevqa.github.io