CountQA: How Well Do MLLMs Count in the Wild?

TL;DR

CountQA introduces a challenging benchmark to evaluate and improve the object counting ability of Multimodal Large Language Models in complex, real-world scenarios, revealing significant performance gaps.

Contribution

This paper presents CountQA, the first comprehensive benchmark for assessing MLLMs' object counting in realistic, cluttered images, highlighting their current limitations and guiding future improvements.

Findings

Top model achieves 42.9% accuracy on CountQA

Performance drops as object counts increase

Benchmark reveals significant counting weaknesses in MLLMs

Abstract

Multimodal Large Language Models (MLLMs) demonstrate remarkable fluency in understanding visual scenes, yet they exhibit a critical lack in a fundamental cognitive skill: object counting. This blind spot severely limits their reliability in real-world applications. To date, this capability has been largely unevaluated in complex scenarios, as existing benchmarks either feature sparse object densities or are confined to specific visual domains, failing to test models under realistic conditions. Addressing this gap, we introduce CountQA, a challenging new benchmark designed to probe this deficiency. Comprising over 1,500 question-answer pairs, CountQA features real-world images with high object density, clutter, and occlusion. We investigate this weakness by evaluating 15 prominent MLLMs on the CountQA benchmark and reveal that the top-performing model achieves a mere 42.9% accuracy, with performance declining as object counts rise. By providing a dedicated benchmark to diagnose and rectify this core weakness, CountQA paves the way for a new generation of MLLMs that are not only descriptively fluent but also numerically grounded and spatially aware. We will open-source the dataset and code upon paper acceptance to foster further research.