CRAFT-LoRA: Content-Style Personalization via Rank-Constrained Adaptation and Training-Free Fusion

TL;DR

CRAFT-LoRA introduces a novel method for personalized image generation that effectively balances content and style by combining rank-constrained fine-tuning, semantic control, and a training-free guidance scheme, improving disentanglement and stability.

Contribution

It proposes a comprehensive framework integrating rank-constrained adaptation, semantic control via expert encoders, and a training-free guidance method for improved content-style personalization.

Findings

Enhanced content-style disentanglement

Flexible semantic control over LoRA modules

High-fidelity generation without retraining

Abstract

Personalized image generation requires effectively balancing content fidelity with stylistic consistency when synthesizing images based on text and reference examples. Low-Rank Adaptation (LoRA) offers an efficient personalization approach, with potential for precise control through combining LoRA weights on different concepts. However, existing combination techniques face persistent challenges: entanglement between content and style representations, insufficient guidance for controlling elements' influence, and unstable weight fusion that often require additional training. We address these limitations through CRAFT-LoRA, with complementary components: (1) rank-constrained backbone fine-tuning that injects low-rank projection residuals to encourage learning decoupled content and style subspaces; (2) a prompt-guided approach featuring an expert encoder with specialized branches that enables semantic extension and precise control through selective adapter aggregation; and (3) a training-free, timestep-dependent classifier-free guidance scheme that enhances generation stability by strategically adjusting noise predictions across diffusion steps. Our method significantly improves content-style disentanglement, enables flexible semantic control over LoRA module combinations, and achieves high-fidelity generation without additional retraining overhead.