WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Models

TL;DR

WorldVQA is a new benchmark that rigorously evaluates the visual world knowledge of multimodal large language models by focusing on their memorization and grounding abilities across a wide range of visual entities.

Contribution

The paper introduces WorldVQA, a benchmark that decouples visual knowledge retrieval from reasoning to accurately measure what models memorize about the visual world.

Findings

WorldVQA effectively measures visual factuality and memorization.

It distinguishes between visual knowledge retrieval and reasoning capabilities.

The benchmark covers a broad spectrum of visual entities from common to rare.

Abstract

We introduce WorldVQA, a benchmark designed to evaluate the atomic visual world knowledge of Multimodal Large Language Models (MLLMs). Unlike current evaluations, which often conflate visual knowledge retrieval with reasoning, WorldVQA decouples these capabilities to strictly measure "what the model memorizes." The benchmark assesses the atomic capability of grounding and naming visual entities across a stratified taxonomy, spanning from common head-class objects to long-tail rarities. We expect WorldVQA to serve as a rigorous test for visual factuality, thereby establishing a standard for assessing the encyclopedic breadth and hallucination rates of current and next-generation frontier models.