AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Models

TL;DR

This paper introduces AEGIS, a comprehensive benchmark for evaluating unified multimodal models' world knowledge across diverse tasks, revealing significant knowledge gaps and the potential of reasoning modules to improve performance.

Contribution

The paper presents AEGIS, a new multi-task benchmark with a novel deterministic evaluation protocol to better assess world knowledge in multimodal models.

Findings

Most UMMs show significant world knowledge deficits.

Performance drops with complex reasoning tasks.

Simple reasoning modules can partially improve UMMs.

Abstract

The capability of Unified Multimodal Models (UMMs) to apply world knowledge across diverse tasks remains a critical, unresolved challenge. Existing benchmarks fall short, offering only siloed, single-task evaluations with limited diagnostic power. To bridge this gap, we propose AEGIS (\emph{i.e.}, \textbf{A}ssessing \textbf{E}diting, \textbf{G}eneration, \textbf{I}nterpretation-Understanding for \textbf{S}uper-intelligence), a comprehensive multi-task benchmark covering visual understanding, generation, editing, and interleaved generation. AEGIS comprises 1,050 challenging, manually-annotated questions spanning 21 topics (including STEM, humanities, daily life, etc.) and 6 reasoning types. To concretely evaluate the performance of UMMs in world knowledge scope without ambiguous metrics, we further propose Deterministic Checklist-based Evaluation (DCE), a protocol that replaces ambiguous prompt-based scoring with atomic ``Y/N'' judgments, to enhance evaluation reliability. Our extensive experiments reveal that most UMMs exhibit severe world knowledge deficits and that performance degrades significantly with complex reasoning. Additionally, simple plug-in reasoning modules can partially mitigate these vulnerabilities, highlighting a promising direction for future research. These results highlight the importance of world-knowledge-based reasoning as a critical frontier for UMMs.