Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning

TL;DR

This paper introduces ViKeR, a novel method for unlearning in multimodal large language models that uses visual cues to identify and prioritize key tokens during the unlearning process, improving effectiveness and coherence.

Contribution

The paper proposes a new visual-guided regularization approach for unlearning in MLLMs, addressing the limitations of existing methods by focusing on key tokens and incorporating visual information.

Findings

Effective unlearning demonstrated on MLLMU and CLEAR benchmarks.

Mitigates forgetting while maintaining response coherence.

Prioritizes key tokens using visual cues and entropy-based measures.

Abstract

Unlearning in Multimodal Large Language Models (MLLMs) prevents the model from revealing private information when queried about target images. Existing MLLM unlearning methods largely adopt approaches developed for LLMs. They treat all answer tokens uniformly, disregarding their varying importance in the unlearning process. Moreover, these methods focus exclusively on the language modality, disregarding visual cues that indicate key tokens in answers. In this paper, after formulating the problem of unlearning in multimodal question answering for MLLMs, we propose Visual-Guided Key-Token Regularization (ViKeR). We leverage irrelevant visual inputs to predict ideal post-unlearning token-level distributions and use these distributions to regularize the unlearning process, thereby prioritizing key tokens. Further, we define key tokens in unlearning via information entropy and discuss ViKeR's effectiveness through token-level gradient reweighting, which amplifies updates on key tokens. Experiments on MLLMU and CLEAR benchmarks demonstrate that our method effectively performs unlearning while mitigating forgetting and maintaining response coherence.