ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

TL;DR

ICU-Bench is a new benchmark for evaluating continual unlearning in multimodal large language models, focusing on privacy-sensitive data and highlighting current methods' limitations in real-world scenarios.

Contribution

The paper introduces ICU-Bench, a comprehensive continual unlearning benchmark with new metrics and extensive experiments revealing the challenges in existing methods.

Findings

Existing unlearning methods struggle in continual settings.

Current methods have limitations in balancing forgetting, utility, and scalability.

ICU-Bench enables detailed analysis of continual unlearning performance.

Abstract

Although Multimodal Large Language Models (MLLMs) have achieved remarkable progress across many domains, their training on large-scale multimodal datasets raises serious privacy concerns, making effective machine unlearning increasingly necessary. However, existing benchmarks mainly focus on static or short-sequence settings, offering limited support for evaluating continual privacy deletion requests in realistic deployments. To bridge this gap, we introduce ICU-Bench, a continual multimodal unlearning benchmark built on privacy-critical document data. ICU-Bench contains 1,000 privacy-sensitive profiles from two document domains, medical reports and labor contracts, with 9,500 images, 16,000 question-answer pairs, and 100 forget tasks. Additionally, new continual unlearning metrics are introduced, facilitating a comprehensive analysis of forgetting effectiveness, historical forgetting preservation, retained utility, and stability throughout the continual unlearning process. Through extensive experiments with representative unlearning methods on ICU-Bench, we show that existing methods generally struggle in continual settings and exhibit clear limitations in balancing forgetting quality, utility preservation, and scalability over long task sequences. These findings highlight the need for multimodal unlearning methods explicitly designed for continual privacy deletion.