Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models

TL;DR

SPACE introduces a novel cross-attention sparsity method for effective concept erasure in large diffusion models, improving robustness and reducing memory use.

Contribution

It proposes a closed-form, sparsity-inducing update for cross-attention parameters that enhances concept erasure in large-scale diffusion models.

Findings

SPACE achieves 80-90% sparsity in cross-attention.

It outperforms dense baselines in erasure effectiveness.

SPACE reduces storage needs by 70%.

Abstract

Erasing specific concepts from text-to-image diffusion models is essential for avoiding the generation of copyrighted and explicit content. Closed-form concept erasure methods offer a fast alternative to backpropagation-based techniques, but they become less effective when scaling from smaller models such as Stable Diffusion 1.5 to larger models like Stable Diffusion XL. To maintain erasure effectiveness in these larger-scale architectures, we propose SParse cross-Attention-based Concept Erasure (SPACE). SPACE iteratively modifies the cross-attention parameters of a model with a closed-form update that jointly induces sparsity and erases target concepts. By concentrating the concept mapping to a lower-dimensional subspace, SPACE achieves superior erasure efficacy compared to dense baselines. Extensive experimental results show improvements in erasure effectiveness and robustness against adversarial prompts. Furthermore, SPACE achieves 80\%-90\% cross-attention sparsity, reducing the storage requirements for saving the modified parameters by 70\%, demonstrating its memory efficiency.