APT: Adaptive Personalized Training for Diffusion Models with Limited Data

TL;DR

This paper introduces Adaptive Personalized Training (APT), a new framework for fine-tuning diffusion models with limited data that reduces overfitting, preserves prior knowledge, and maintains semantic coherence.

Contribution

APT employs adaptive training, feature regularization, and attention alignment to improve personalization of diffusion models with scarce data.

Findings

APT reduces overfitting during fine-tuning.

APT maintains semantic coherence and prior knowledge.

APT outperforms existing methods in limited-data scenarios.

Abstract

Personalizing diffusion models using limited data presents significant challenges, including overfitting, loss of prior knowledge, and degradation of text alignment. Overfitting leads to shifts in the noise prediction distribution, disrupting the denoising trajectory and causing the model to lose semantic coherence. In this paper, we propose Adaptive Personalized Training (APT), a novel framework that mitigates overfitting by employing adaptive training strategies and regularizing the model's internal representations during fine-tuning. APT consists of three key components: (1) Adaptive Training Adjustment, which introduces an overfitting indicator to detect the degree of overfitting at each time step bin and applies adaptive data augmentation and adaptive loss weighting based on this indicator; (2)Representation Stabilization, which regularizes the mean and variance of intermediate feature maps to prevent excessive shifts in noise prediction; and (3) Attention Alignment for Prior Knowledge Preservation, which aligns the cross-attention maps of the fine-tuned model with those of the pretrained model to maintain prior knowledge and semantic coherence. Through extensive experiments, we demonstrate that APT effectively mitigates overfitting, preserves prior knowledge, and outperforms existing methods in generating high-quality, diverse images with limited reference data.