HAM: A Training-Free Style Transfer Approach via Heterogeneous Attention Modulation for Diffusion Models

TL;DR

This paper introduces HAM, a training-free style transfer method for diffusion models that uses heterogeneous attention modulation to better preserve content identity while transferring complex styles.

Contribution

It proposes a novel training-free approach employing heterogeneous attention modulation, including GAR and LAT, to improve style transfer quality and content preservation in diffusion models.

Findings

Achieves state-of-the-art performance on multiple metrics.

Effectively preserves content identity during style transfer.

Handles complex style references better than existing methods.

Abstract

Diffusion models have demonstrated remarkable performance in image generation, particularly within the domain of style transfer. Prevailing style transfer approaches typically leverage pre-trained diffusion models' robust feature extraction capabilities alongside external modular control pathways to explicitly impose style guidance signals. However, these methods often fail to capture complex style reference or retain the identity of user-provided content images, thus falling into the trap of style-content balance. Thus, we propose a training-free style transfer approach via $\textbf{h}$eterogeneous $\textbf{a}$ttention $\textbf{m}$odulation ($\textbf{HAM}$) to protect identity information during image/text-guided style reference transfer, thereby addressing the style-content trade-off challenge. Specifically, we first introduces style noise initialization to initialize latent noise for diffusion. Then, during the diffusion process, it innovatively employs HAM for different attention mechanisms, including Global Attention Regulation (GAR) and Local Attention Transplantation (LAT), which better preserving the details of the content image while capturing complex style references. Our approach is validated through a series of qualitative and quantitative experiments, achieving state-of-the-art performance on multiple quantitative metrics.