# The Percept-V Challenge: Can Multimodal LLMs Crack Simple Perception Problems?

## TL;DR

This paper introduces Percept-V, a dataset designed to evaluate multimodal large language models on simple visual perception tasks, revealing that current models perform poorly compared to humans, especially as image complexity increases.

## Contribution

The paper presents Percept-V, a novel dataset for assessing basic perception skills in MLLMs, highlighting their limitations on simple perception tasks.

## Key findings

- MLLMs perform poorly compared to humans on Percept-V.
- Performance declines rapidly as the number of objects increases.
- Certain perception skills are significantly harder for all models.

## Abstract

Cognitive science research treats visual perception, the ability to understand and make sense of a visual input, as one of the early developmental signs of intelligence. Its TVPS-4 framework categorizes and tests human perception into seven skills such as visual discrimination, and form constancy. Do Multimodal Large Language Models (MLLMs) match up to humans in basic perception? Even though there are many benchmarks that evaluate MLLMs on advanced reasoning and knowledge skills, there is limited research that focuses evaluation on simple perception. In response, we introduce Percept-V, a dataset containing 6000 program-generated uncontaminated images divided into 30 domains, where each domain tests one or more TVPS-4 skills. Our focus is on perception, so we make our domains quite simple and the reasoning and knowledge required for solving them are minimal. Since modern-day MLLMs can solve much more complex tasks, our a-priori expectation is that they will solve these domains very easily. Contrary to our belief, our experiments show a weak performance of SoTA proprietary and open-source MLLMs compared to very high human performance on Percept-V. We find that as number of objects in the image increases, performance goes down rather fast. Our experiments also identify the perception skills that are considerably harder for all models.