$Z^*$: Zero-shot Style Transfer via Attention Rearrangement

TL;DR

This paper introduces Z-STAR, a zero-shot style transfer method using diffusion models and attention rearrangement, enabling style transfer without retraining and addressing attention blending issues.

Contribution

The study presents a novel zero-shot style transfer approach leveraging diffusion models and a cross-attention rearrangement strategy, without requiring retraining.

Findings

Effective style transfer demonstrated with diffusion models.

Attention rearrangement improves content preservation.

Outperforms existing methods in quality and flexibility.

Abstract

Despite the remarkable progress in image style transfer, formulating style in the context of art is inherently subjective and challenging. In contrast to existing learning/tuning methods, this study shows that vanilla diffusion models can directly extract style information and seamlessly integrate the generative prior into the content image without retraining. Specifically, we adopt dual denoising paths to represent content/style references in latent space and then guide the content image denoising process with style latent codes. We further reveal that the cross-attention mechanism in latent diffusion models tends to blend the content and style images, resulting in stylized outputs that deviate from the original content image. To overcome this limitation, we introduce a cross-attention rearrangement strategy. Through theoretical analysis and experiments, we demonstrate the effectiveness and superiority of the diffusion-based $\underline{Z}$ero-shot $\underline{S}$tyle $\underline{T}$ransfer via $\underline{A}$ttention $\underline{R}$earrangement, Z-STAR.