SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

TL;DR

SALMUBench is a new benchmark for evaluating fine-grained unlearning of sensitive association data in multimodal models, highlighting current methods' limitations and providing tools for future research.

Contribution

Introduces SALMUBench, a synthetic dataset and evaluation protocol for association-level unlearning in multimodal models, with publicly available resources.

Findings

Current unlearning methods either fail to forget effectively or over-generalize.

SALMUBench reveals distinct failure modes in existing unlearning techniques.

Benchmark sets new standards for comprehensive evaluation of unlearning methods.

Abstract

As multimodal models like CLIP become integral to downstream systems, the need to remove sensitive information is critical. However, machine unlearning for contrastively-trained encoders remains underexplored, and existing evaluations fail to diagnose fine-grained, association-level forgetting. We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations and two foundational models: a Compromised model polluted with this data, and a Clean model without it. To isolate unlearning effects, both are trained from scratch on the same 400M-pair retain base, with the Compromised model additionally trained on the sensitive set. We propose a novel evaluation protocol with structured holdout sets (holdout identity, holdout association) to precisely measure unlearning efficacy and collateral damage. Our benchmark reveals that while utility-efficient deletion is feasible, current methods exhibit distinct failure modes: they either fail to forget effectively or over-generalize by erasing more than intended. SALMUBench sets a new standard for comprehensive unlearning evaluation, and we publicly release our dataset, models, evaluation scripts, and leaderboards to foster future research.