MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge

TL;DR

This paper introduces MMKU-Bench, a comprehensive benchmark for evaluating how multimodal models update and maintain knowledge, addressing limitations of previous methods and analyzing cross-modal consistency.

Contribution

It presents MMKU-Bench, a large-scale benchmark for multimodal knowledge updating, and evaluates various approaches, revealing strengths and limitations in knowledge preservation and updating.

Findings

Supervised fine-tuning and RLHF tend to forget previously learned knowledge.

Knowledge editing better preserves capabilities but struggles with continual updates.

MMKU-Bench enables systematic analysis of cross-modal knowledge consistency.

Abstract

As real-world knowledge continues to evolve, the parametric knowledge acquired by multimodal models during pretraining becomes increasingly difficult to remain consistent with real-world knowledge. Existing research on multimodal knowledge updating focuses only on learning previously unknown knowledge, while overlooking the need to update knowledge that the model has already mastered but that later changes; moreover, evaluation is limited to the same modality, lacking a systematic analysis of cross-modal consistency. To address these issues, this paper proposes MMKU-Bench, a comprehensive evaluation benchmark for multimodal knowledge updating, which contains over 25k knowledge instances and more than 49k images, covering two scenarios, updated knowledge and unknown knowledge, thereby enabling comparative analysis of learning across different knowledge types. On this benchmark, we evaluate a variety of representative approaches, including supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), and knowledge editing (KE). Experimental results show that SFT and RLHF are prone to catastrophic forgetting, while KE better preserve general capabilities but exhibit clear limitations in continual updating. Overall, MMKU-Bench provides a reliable and comprehensive evaluation benchmark for multimodal knowledge updating, advancing progress in this field.