Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models

TL;DR

This paper introduces CoVUBench, a benchmark for evaluating how well large vision-language models can unlearn copyrighted visual content, addressing a critical challenge in responsible AI deployment.

Contribution

It presents the first systematic framework and evaluation protocol for measuring copyright content unlearning effectiveness in multimodal large models.

Findings

CoVUBench enables robust assessment of unlearning in LVLMs.

The benchmark balances copyright content removal with model utility preservation.

Procedurally generated synthetic data ensures realistic evaluation scenarios.

Abstract

Large Vision-Language Models (LVLMs), trained on web-scale data, risk memorizing and regenerating copyrighted visual content such as characters and logos, creating significant challenges. Machine unlearning offers a path to mitigate these risks by removing specific content post-training, but evaluating its effectiveness, especially in the complex multimodal setting of LVLMs, remains an open problem. Current evaluation methods often lack robustness or fail to capture the nuances of cross-modal concept erasure. To address this critical gap, we introduce the CoVUBench benchmark, the first framework specifically designed for evaluating copyright content unlearning in LVLMs. CoVUBench utilizes procedurally generated, legally safe synthetic data coupled with systematic visual variations spanning compositional changes and diverse domain manifestations to ensure realistic and robust evaluation of unlearning generalization. Our comprehensive multimodal evaluation protocol assesses both forgetting efficacy from the copyright holder perspective and the preservation of general model utility from the deployer viewpoint. By rigorously measuring this crucial trade-off, CoVUBench provides a standardized tool to advance the development of responsible and effective unlearning methods for LVLMs.