Break Stylistic Sophon: Are We Really Meant to Confine the Imagination in Style Transfer?

TL;DR

This paper introduces StyleWallfacer, a unified framework for style transfer that combines semantic style injection, data augmentation with human feedback, and a training-free diffusion process to achieve high-quality, artist-level style transfer and text-driven stylization, including color editing.

Contribution

The paper presents a novel unified framework that integrates semantic style injection, human feedback-based data augmentation, and a diffusion process for advanced style transfer and stylization.

Findings

Achieved artist-level style transfer results.

Enabled image color editing during style transfer.

Reduced overfitting with feedback-driven data augmentation.

Abstract

In this pioneering study, we introduce StyleWallfacer, a groundbreaking unified training and inference framework, which not only addresses various issues encountered in the style transfer process of traditional methods but also unifies the framework for different tasks. This framework is designed to revolutionize the field by enabling artist level style transfer and text driven stylization. First, we propose a semantic-based style injection method that uses BLIP to generate text descriptions strictly aligned with the semantics of the style image in CLIP space. By leveraging a large language model to remove style-related descriptions from these descriptions, we create a semantic gap. This gap is then used to fine-tune the model, enabling efficient and drift-free injection of style knowledge. Second, we propose a data augmentation strategy based on human feedback, incorporating high-quality samples generated early in the fine-tuning process into the training set to facilitate progressive learning and significantly reduce its overfitting. Finally, we design a training-free triple diffusion process using the fine-tuned model, which manipulates the features of self-attention layers in a manner similar to the cross-attention mechanism. Specifically, in the generation process, the key and value of the content-related process are replaced with those of the style-related process to inject style while maintaining text control over the model. We also introduce query preservation to mitigate disruptions to the original content. Under such a design, we have achieved high-quality image-driven style transfer and text-driven stylization, delivering artist-level style transfer results while preserving the original image content. Moreover, we achieve image color editing during the style transfer process for the first time.