VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

TL;DR

This paper introduces VKnowU, a benchmark for evaluating visual knowledge in multimodal large language models, revealing current models' limitations and proposing a new dataset and baseline to enhance understanding of physical and social worlds.

Contribution

The paper presents VKnowU, a comprehensive benchmark for visual knowledge, and introduces VideoKnow+, a baseline model that improves understanding of physical and social principles in MLLMs.

Findings

Leading models underperform compared to humans in visual knowledge tasks.

VideoKnow+ achieves a +3.7% improvement on VKnowU benchmark.

Visual knowledge is crucial for more generalizable multimodal models.

Abstract

While Multimodal Large Language Models (MLLMs) have become adept at recognizing objects, they often lack the intuitive, human-like understanding of the world's underlying physical and social principles. This high-level vision-grounded semantics, which we term visual knowledge, forms a bridge between perception and reasoning, yet remains an underexplored area in current MLLMs. To systematically evaluate this capability, we present VKnowU, a comprehensive benchmark featuring 1,680 questions in 1,249 videos, covering 8 core types of visual knowledge spanning both world-centric (e.g., intuitive physics) and human-centric (e.g., subjective intentions). Evaluation of 23 SOTA MLLMs reveals that leading models still fall short of human performance, with particularly notable gaps in the world-centric. To bridge this gap, we introduce a new dataset, VKnowQA, and VideoKnow+, a baseline model that explicitly incorporates visual knowledge into MLLMs. VideoKnow+ follows a structured See-Think-Answer paradigm and adopts reinforcement learning with visual knowledge reward, achieving a +3.7% improvement on VKnowU and consistent gains on MVBench, Video-MME, and MMVU. Our work highlights visual knowledge as a missing cornerstone for developing more generalizable MLLMs that can not only see but also truly understand our physical and social worlds.