Rethinking the Vulnerability of Concept Erasure and a New Method

TL;DR

This paper analyzes the vulnerabilities in concept erasure methods for diffusion models, revealing pervasive prompt embedding weaknesses and introducing RECORD, a superior restoration algorithm that significantly outperforms existing methods.

Contribution

It uncovers fundamental vulnerabilities in concept-erased models and proposes RECORD, a novel, highly effective restoration algorithm with improved performance.

Findings

Vulnerabilities are widespread in prompt embedding spaces of erased models.

RECORD outperforms existing restoration methods by up to 17.8 times.

Proposed acceleration strategies improve compute-performance tradeoff.

Abstract

The proliferation of text-to-image diffusion models has raised significant privacy and security concerns, particularly regarding the generation of copyrighted or harmful images. In response, concept erasure (defense) methods have been developed to "unlearn" specific concepts through post-hoc finetuning. However, recent concept restoration (attack) methods have demonstrated that these supposedly erased concepts can be recovered using adversarially crafted prompts, revealing a critical vulnerability in current defense mechanisms. In this work, we first investigate the fundamental sources of adversarial vulnerability and reveal that vulnerabilities are pervasive in the prompt embedding space of concept-erased models, a characteristic inherited from the original pre-unlearned model. Furthermore, we introduce **RECORD**, a novel coordinate-descent-based restoration algorithm that consistently outperforms existing restoration methods by up to 17.8 times. We conduct extensive experiments to assess its compute-performance tradeoff and propose acceleration strategies.