CLEAR: Character Unlearning in Textual and Visual Modalities

TL;DR

CLEAR introduces the first open benchmark for multimodal unlearning, enabling evaluation of methods that remove private information from combined text and visual data, highlighting the importance of joint modality unlearning.

Contribution

This paper presents CLEAR, the first open-source benchmark for multimodal unlearning, facilitating systematic evaluation and comparison of unlearning methods across text and visual modalities.

Findings

Joint unlearning of text and images outperforms single-modality approaches.

Evaluation of 11 MU methods across four datasets shows varying effectiveness.

CLEAR enables comprehensive assessment of multimodal unlearning techniques.

Abstract

Machine Unlearning (MU) is critical for removing private or hazardous information from deep learning models. While MU has advanced significantly in unimodal (text or vision) settings, multimodal unlearning (MMU) remains underexplored due to the lack of open benchmarks for evaluating cross-modal data removal. To address this gap, we introduce CLEAR, the first open-source benchmark designed specifically for MMU. CLEAR contains 200 fictitious individuals and 3,700 images linked with corresponding question-answer pairs, enabling a thorough evaluation across modalities. We conduct a comprehensive analysis of 11 MU methods (e.g., SCRUB, gradient ascent, DPO) across four evaluation sets, demonstrating that jointly unlearning both modalities outperforms single-modality approaches. The dataset is available at https://huggingface.co/datasets/therem/CLEAR