Erasure or Erosion? Evaluating Compositional Degradation in Unlearned Text-To-Image Diffusion Models

TL;DR

This paper systematically evaluates how different unlearning methods affect the ability of text-to-image diffusion models to generate images with complex compositions, revealing a trade-off between erasure effectiveness and generative integrity.

Contribution

It provides a comprehensive empirical analysis of concept unlearning in diffusion models, highlighting the limitations and trade-offs in current approaches.

Findings

Strong erasure often degrades compositional capabilities.

Preserving composition typically reduces unlearning effectiveness.

Current evaluation metrics may not fully capture semantic preservation.

Abstract

Post-hoc unlearning has emerged as a practical mechanism for removing undesirable concepts from large text-to-image diffusion models. However, prior work primarily evaluates unlearning through erasure success; its impact on broader generative capabilities remains poorly understood. In this work, we conduct a systematic empirical study of concept unlearning through the lens of compositional text-to-image generation. Focusing on nudity removal in Stable Diffusion 1.4, we evaluate a diverse set of state-of-the-art unlearning methods using T2I-CompBench++ and GenEval, alongside established unlearning benchmarks. Our results reveal a consistent trade-off between unlearning effectiveness and compositional integrity: methods that achieve strong erasure frequently incur substantial degradation in attribute binding, spatial reasoning, and counting. Conversely, approaches that preserve compositional structure often fail to provide robust erasure. These findings highlight limitations of current evaluation practices and underscore the need for unlearning objectives that explicitly account for semantic preservation beyond targeted suppression.