Z-STAR+: A Zero-shot Style Transfer Method via Adjusting Style Distribution

TL;DR

Z-STAR+ introduces a diffusion-based zero-shot style transfer method that directly utilizes latent features from diffusion models, enabling flexible, artifact-free stylization without retraining.

Contribution

The paper presents a novel zero-shot style transfer approach leveraging diffusion models' latent features and a new style distribution adjustment mechanism, surpassing traditional style representations.

Findings

Effective style transfer without retraining.

Superior style alignment and artifact reduction.

Outperforms existing methods in quality and flexibility.

Abstract

Style transfer presents a significant challenge, primarily centered on identifying an appropriate style representation. Conventional methods employ style loss, derived from second-order statistics or contrastive learning, to constrain style representation in the stylized result. However, these pre-defined style representations often limit stylistic expression, leading to artifacts. In contrast to existing approaches, we have discovered that latent features in vanilla diffusion models inherently contain natural style and content distributions. This allows for direct extraction of style information and seamless integration of generative priors into the content image without necessitating retraining. Our method adopts dual denoising paths to represent content and style references in latent space, subsequently guiding the content image denoising process with style latent codes. We introduce a Cross-attention Reweighting module that utilizes local content features to query style image information best suited to the input patch, thereby aligning the style distribution of the stylized results with that of the style image. Furthermore, we design a scaled adaptive instance normalization to mitigate inconsistencies in color distribution between style and stylized images on a global scale. Through theoretical analysis and extensive experimentation, we demonstrate the effectiveness and superiority of our diffusion-based \uline{z}ero-shot \uline{s}tyle \uline{t}ransfer via \uline{a}djusting style dist\uline{r}ibution, termed Z-STAR+.