Beyond Memorization: Selective Learning for Copyright-Safe Diffusion Model Training

TL;DR

This paper presents a gradient projection technique for diffusion models that selectively excludes sensitive features during training, significantly reducing memorization of proprietary data while maintaining high-quality image generation.

Contribution

The authors introduce a novel gradient projection method for concept-level feature exclusion, enhancing privacy and IP safety in diffusion model training without sacrificing performance.

Findings

Reduces memorization of sensitive features by over 90%

Maintains image quality and semantic fidelity

Seamlessly integrates with existing training pipelines

Abstract

Memorization in large-scale text-to-image diffusion models poses significant security and intellectual property risks, enabling adversarial attribute extraction and the unauthorized reproduction of sensitive or proprietary features. While conventional dememorization techniques, such as regularization and data filtering, limit overfitting to specific training examples, they fail to systematically prevent the internalization of prohibited concept-level features. Simply discarding all images containing a sensitive feature wastes invaluable training data, necessitating a method for selective learning at the concept level. We introduce a gradient projection method designed to enforce a stringent requirement of concept-level feature exclusion. Our defense operates during backpropagation by systematically identifying and excising training signals aligned with embeddings of prohibited attributes. Specifically, we project each gradient update onto the orthogonal complement of the sensitive feature's embedding space, thereby zeroing out its influence on the model's weights. Our method integrates seamlessly into standard diffusion model training pipelines and complements existing defenses. We analyze our method against an adversary aiming for feature extraction. In extensive experiments, we demonstrate that our framework drastically reduces memorization while rigorously preserving generation quality and semantic fidelity. By reframing memorization control as selective learning, our approach establishes a new paradigm for IP-safe and privacy-preserving generative AI.