Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearning

TL;DR

This paper introduces a comprehensive benchmark for evaluating unlearning methods in text-to-image diffusion models across multiple dimensions, addressing broader impacts beyond concept removal quality.

Contribution

It proposes the Holistic Unlearning Benchmark (HUB), a multi-faceted evaluation framework covering six key criteria for assessing unlearning methods.

Findings

No single method excels across all criteria

The benchmark covers 33 concepts with 16,000 prompts each

Releasing code and dataset to foster further research

Abstract

As text-to-image diffusion models gain widespread commercial applications, there are increasing concerns about unethical or harmful use, including the unauthorized generation of copyrighted or sensitive content. Concept unlearning has emerged as a promising solution to these challenges by removing undesired and harmful information from the pre-trained model. However, the previous evaluations primarily focus on whether target concepts are removed while preserving image quality, neglecting the broader impacts such as unintended side effects. In this work, we propose Holistic Unlearning Benchmark (HUB), a comprehensive framework for evaluating unlearning methods across six key dimensions: faithfulness, alignment, pinpoint-ness, multilingual robustness, attack robustness, and efficiency. Our benchmark covers 33 target concepts, including 16,000 prompts per concept, spanning four categories: Celebrity, Style, Intellectual Property, and NSFW. Our investigation reveals that no single method excels across all evaluation criteria. By releasing our evaluation code and dataset, we hope to inspire further research in this area, leading to more reliable and effective unlearning methods.