KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context

TL;DR

KMMMU is a Korean multimodal benchmark with 3,466 culturally and discipline-specific questions designed to evaluate AI understanding in Korean contexts, revealing significant performance gaps and challenges.

Contribution

This paper introduces KMMMU, the first Korean-specific multimodal benchmark focusing on local conventions and discipline-specific visual formats for AI evaluation.

Findings

Open-source models achieve only 42.05% accuracy on the full set.

Proprietary models reach 52.42% accuracy on the hard subset.

Performance varies across disciplines, highlighting domain-specific challenges.

Abstract

We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings. KMMMU contains 3,466 questions from exams natively written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions. Unlike translated or English-centric benchmarks, KMMMU targets information-dense problems shaped by local conventions, official standards, and discipline-specific visual formats. Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset. Performance varies across disciplines, with some disciplines emerging as bottlenecks, and Korean-specific questions showing gaps of up to 13.43%. Error analysis suggests that these failures stem less from insufficient reasoning depth than from weak convention-to-label mapping, few-shot symbolic induction, localized knowledge recall, and domain-specific standards understanding. KMMMU provides a testbed for multimodal evaluation beyond English-centric benchmarks and for developing more reliable systems for expert real-world tasks.