When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations

TL;DR

This paper introduces a new benchmark for evaluating large multimodal models' ability to incorporate evolving knowledge and proposes methods to improve knowledge injection and retention.

Contribution

It presents MMEVOKE, a benchmark for multimodal evolving knowledge, and explores new techniques for knowledge augmentation and retention in LMMs.

Findings

Knowledge-aware augmentation improves injection performance.

Data Replay and MoE methods reduce capability degradation.

Existing methods face challenges in dynamic knowledge injection.

Abstract

Large Multimodal Models (LMMs) store vast amounts of pretrained knowledge but struggle to remain aligned with real-world updates, making it difficult to avoid capability degradation when acquiring evolving knowledge. Furthermore, most current work focuses on exploring static textual knowledge injection, neglecting dynamic multimodal evolving knowledge injection, leaving the potential of LMMs for multimodal knowledge injection as an open question. To address this, we first propose a pipeline to construct MMEVOKE, a benchmark for evaluating LMMs' ability in multimodal evolving knowledge injection. MMEVOKE contains 9,422 samples spanning 159 subtypes. Then, based on extensive experiments with MMEVOKE, we reveal challenges such as poor injection performance and capability degradation in existing knowledge injection methods through knowledge injection tests and general capability tests. Finally, to tackle these challenges, we introduce knowledge augmentation and knowledge retention methods, finding that knowledge-aware augmentation strengthens knowledge injection performance, and that Data Replay and MoE methods effectively mitigate capability degradation.