Robust MLLM Unlearning via Visual Knowledge Distillation

TL;DR

This paper introduces a novel visual knowledge distillation method for unlearning specific visual information in multimodal large language models, improving effectiveness, efficiency, and robustness against attacks.

Contribution

It proposes a new MLLM unlearning approach that disentangles visual and textual knowledge, focusing on visual component fine-tuning with intermediate visual supervision.

Findings

Outperforms state-of-the-art unlearning methods in effectiveness and efficiency

Enhances robustness of MLLM unlearning against relearning attacks

Efficiently erases visual knowledge while preserving textual knowledge

Abstract

Recently, machine unlearning approaches have been proposed to remove sensitive information from well-trained large models. However, most existing methods are tailored for LLMs, while MLLM-oriented unlearning remains at its early stage. Inspired by recent studies exploring the internal mechanisms of MLLMs, we propose to disentangle the visual and textual knowledge embedded within MLLMs and introduce a dedicated approach to selectively erase target visual knowledge while preserving textual knowledge. Unlike previous unlearning methods that rely on output-level supervision, our approach introduces a Visual Knowledge Distillation (VKD) scheme, which leverages intermediate visual representations within the MLLM as supervision signals. This design substantially enhances both unlearning effectiveness and model utility. Moreover, since our method only fine-tunes the visual components of the MLLM, it offers significant efficiency advantages. Extensive experiments demonstrate that our approach outperforms state-of-the-art unlearning methods in terms of both effectiveness and efficiency. Moreover, we are the first to evaluate the robustness of MLLM unlearning against relearning attacks.