Can VLMs Truly Forget? Benchmarking Training-Free Visual Concept Unlearning

TL;DR

This paper introduces VLM-UnBench, a comprehensive benchmark for evaluating training-free visual concept unlearning in vision-language models, revealing limitations of current suppression methods.

Contribution

It provides the first rigorous benchmark for training-free visual concept unlearning, analyzing effectiveness across multiple datasets, concepts, and model configurations.

Findings

Realistic prompts do not significantly forget concepts compared to no-instruction baseline.

Meaningful forgetting only occurs under oracle conditions with explicit concept disclosure.

Object and scene concepts are highly resistant to suppression, even with explicit instructions.

Abstract

VLMs trained on web-scale data retain sensitive and copyrighted visual concepts that deployment may require removing. Training-based unlearning methods share a structural flaw: fine-tuning on a narrow forget set degrades general capabilities before unlearning begins, making it impossible to attribute subsequent performance drops to the unlearning procedure itself. Training-free approaches sidestep this by suppressing concepts through prompts or system instructions, but no rigorous benchmark exists for evaluating them on visual tasks. We introduce VLM-UnBench, the first benchmark for training-free visual concept unlearning in VLMs. It covers four forgetting levels, 7 source datasets, and 11 concept axes, and pairs a three-level probe taxonomy with five evaluation conditions to separate genuine forgetting from instruction compliance. Across 8 evaluation settings and 13 VLM configurations, realistic unlearning prompts leave forget accuracy near the no-instruction baseline; meaningful reductions appear only under oracle conditions that disclose the target concept to the model. Object and scene concepts are the most resistant to suppression, and stronger instruction-tuned models remain capable despite explicit forget instructions. These results expose a clear gap between prompt-level suppression and true visual concept erasure.