Do Concept Replacement Techniques Really Erase Unacceptable Concepts?

TL;DR

This paper investigates whether current concept replacement techniques effectively remove unacceptable concepts in generative models, especially in image-to-image scenarios, and proposes a new method to improve both removal and preservation of desired content.

Contribution

The paper empirically shows CRTs fail to erase unacceptable concepts in I2I models and introduces AntiMirror, a targeted editing technique that enhances both concept removal and fidelity.

Findings

State-of-the-art CRTs do not effectively erase unacceptable concepts in I2I models.

Existing CRTs are less effective in I2I scenarios compared to T2I.

AntiMirror improves concept removal and preserves other input concepts.

Abstract

Generative models, particularly diffusion-based text-to-image (T2I) models, have demonstrated astounding success. However, aligning them to avoid generating content with unacceptable concepts (e.g., offensive or copyrighted content, or celebrity likenesses) remains a significant challenge. Concept replacement techniques (CRTs) aim to address this challenge, often by trying to "erase" unacceptable concepts from models. Recently, model providers have started offering image editing services which accept an image and a text prompt as input, to produce an image altered as specified by the prompt. These are known as image-to-image (I2I) models. In this paper, we first use an I2I model to empirically demonstrate that today's state-of-the-art CRTs do not in fact erase unacceptable concepts. Existing CRTs are thus likely to be ineffective in emerging I2I scenarios, despite their proven ability to remove unwanted concepts in T2I pipelines, highlighting the need to understand this discrepancy between T2I and I2I settings. Next, we argue that a good CRT, while replacing unacceptable concepts, should preserve other concepts specified in the inputs to generative models. We call this fidelity. Prior work on CRTs have neglected fidelity in the case of unacceptable concepts. Finally, we propose the use of targeted image-editing techniques to achieve both effectiveness and fidelity. We present such a technique, AntiMirror, and demonstrate its viability.