Why Instruction-Based Unlearning Fails in Diffusion Models?

TL;DR

This paper demonstrates that instruction-based unlearning methods fail to suppress targeted concepts in diffusion models, revealing a fundamental limitation of current prompt-level unlearning techniques.

Contribution

The study provides a systematic analysis showing the failure of instruction-based unlearning in diffusion models and identifies the underlying reasons related to attention dynamics.

Findings

Unlearning instructions do not reduce attention to targeted concepts in diffusion models.

Targeted concepts persist throughout generation despite unlearning prompts.

Effective unlearning likely requires interventions beyond inference-time language control.

Abstract

Instruction-based unlearning has proven effective for modifying the behavior of large language models at inference time, but whether this paradigm extends to other generative models remains unclear. In this work, we investigate instruction-based unlearning in diffusion-based image generation models and show, through controlled experiments across multiple concepts and prompt variants, that diffusion models systematically fail to suppress targeted concepts when guided solely by natural-language unlearning instructions. By analyzing both the CLIP text encoder and cross-attention dynamics during the denoising process, we find that unlearning instructions do not induce sustained reductions in attention to the targeted concept tokens, causing the targeted concept representations to persist throughout generation. These results reveal a fundamental limitation of prompt-level instruction in diffusion models and suggest that effective unlearning requires interventions beyond inference-time language control.