Targeted Data Protection for Diffusion Model by Matching Training Trajectory

TL;DR

This paper introduces TAFAP, a novel method for targeted data protection in diffusion models, enabling persistent, controllable, and verifiable transformations during training to safeguard against unauthorized data use.

Contribution

TAFAP is the first approach to control entire training trajectories for effective, stable targeted data protection in diffusion models, surpassing snapshot-based methods.

Findings

TAFAP achieves successful targeted transformations in diffusion models.

The method maintains high image quality while redirecting towards target concepts.

TAFAP outperforms existing TDP methods in controllability and robustness.

Abstract

Recent advancements in diffusion models have made fine-tuning text-to-image models for personalization increasingly accessible, but have also raised significant concerns regarding unauthorized data usage and privacy infringement. Current protection methods are limited to passively degrading image quality, failing to achieve stable control. While Targeted Data Protection (TDP) offers a promising paradigm for active redirection toward user-specified target concepts, existing TDP attempts suffer from poor controllability due to snapshot-matching approaches that fail to account for complete learning dynamics. We introduce TAFAP (Trajectory Alignment via Fine-tuning with Adversarial Perturbations), the first method to successfully achieve effective TDP by controlling the entire training trajectory. Unlike snapshot-based methods whose protective influence is easily diluted as training progresses, TAFAP employs trajectory-matching inspired by dataset distillation to enforce persistent, verifiable transformations throughout fine-tuning. We validate our method through extensive experiments, demonstrating the first successful targeted transformation in diffusion models with simultaneous control over both identity and visual patterns. TAFAP significantly outperforms existing TDP attempts, achieving robust redirection toward target concepts while maintaining high image quality. This work enables verifiable safeguards and provides a new framework for controlling and tracing alterations in diffusion model outputs.