Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench

TL;DR

This paper introduces MLLMU-Bench, a new benchmark for evaluating the effectiveness of unlearning algorithms in multimodal large language models, addressing privacy concerns by assessing how models forget sensitive data.

Contribution

The paper presents MLLMU-Bench, the first comprehensive benchmark for multimodal model unlearning, including diverse profiles and evaluation metrics for efficacy, generalizability, and utility.

Findings

Unimodal unlearning algorithms perform well in generation and cloze tasks.

Multimodal unlearning approaches excel in classification tasks with multimodal inputs.

Baseline results demonstrate varying effectiveness across different unlearning algorithms.

Abstract

Generative models such as Large Language Models (LLM) and Multimodal Large Language models (MLLMs) trained on massive web corpora can memorize and disclose individuals' confidential and private data, raising legal and ethical concerns. While many previous works have addressed this issue in LLM via machine unlearning, it remains largely unexplored for MLLMs. To tackle this challenge, we introduce Multimodal Large Language Model Unlearning Benchmark (MLLMU-Bench), a novel benchmark aimed at advancing the understanding of multimodal machine unlearning. MLLMU-Bench consists of 500 fictitious profiles and 153 profiles for public celebrities, each profile feature over 14 customized question-answer pairs, evaluated from both multimodal (image+text) and unimodal (text) perspectives. The benchmark is divided into four sets to assess unlearning algorithms in terms of efficacy, generalizability, and model utility. Finally, we provide baseline results using existing generative model unlearning algorithms. Surprisingly, our experiments show that unimodal unlearning algorithms excel in generation and cloze tasks, while multimodal unlearning approaches perform better in classification tasks with multimodal inputs.