Classifier-Free Guidance inside the Attraction Basin May Cause Memorization

TL;DR

This paper identifies attraction basins in diffusion models as a cause of memorization and proposes a guidance strategy to steer away from these basins, reducing memorization while maintaining image quality.

Contribution

The paper introduces a novel perspective on diffusion model memorization and proposes a guidance method that mitigates memorization by avoiding attraction basins during denoising.

Findings

Attraction basins cause memorization in diffusion models.

Guidance strategies can steer diffusion away from memorization.

Proposed methods effectively reduce memorization while preserving image quality.

Abstract

Diffusion models are prone to exactly reproduce images from the training data. This exact reproduction of the training data is concerning as it can lead to copyright infringement and/or leakage of privacy-sensitive information. In this paper, we present a novel perspective on the memorization phenomenon and propose a simple yet effective approach to mitigate it. We argue that memorization occurs because of an attraction basin in the denoising process which steers the diffusion trajectory towards a memorized image. However, this can be mitigated by guiding the diffusion trajectory away from the attraction basin by not applying classifier-free guidance until an ideal transition point occurs from which classifier-free guidance is applied. This leads to the generation of non-memorized images that are high in image quality and well-aligned with the conditioning mechanism. To further improve on this, we present a new guidance technique, opposite guidance, that escapes the attraction basin sooner in the denoising process. We demonstrate the existence of attraction basins in various scenarios in which memorization occurs, and we show that our proposed approach successfully mitigates memorization.