ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing

TL;DR

ArtCrafter introduces a novel text-to-image style transfer framework that uses attention-based style extraction and embedding reframing to improve stylistic intensity, controllability, and diversity in generated images.

Contribution

The paper presents a new framework combining attention-based style extraction and embedding reframing for improved text-image style transfer.

Findings

Achieves high stylistic intensity and diversity in stylized images.

Enhances controllability over style transfer results.

Demonstrates superior performance through extensive experiments.

Abstract

Recent years have witnessed significant advancements in text-guided style transfer, primarily attributed to innovations in diffusion models. These models excel in conditional guidance, utilizing text or images to direct the sampling process. However, despite their capabilities, direct conditional guidance approaches often face challenges in balancing the expressiveness of textual semantics with the diversity of output results while capturing stylistic features. To address these challenges, we introduce ArtCrafter, a novel framework for text-to-image style transfer. Specifically, we introduce an attention-based style extraction module, meticulously engineered to capture the subtle stylistic elements within an image. This module features a multi-layer architecture that leverages the capabilities of perceiver attention mechanisms to integrate fine-grained information. Additionally, we present a novel text-image aligning augmentation component that adeptly balances control over both modalities, enabling the model to efficiently map image and text embeddings into a shared feature space. We achieve this through attention operations that enable smooth information flow between modalities. Lastly, we incorporate an explicit modulation that seamlessly blends multimodal enhanced embeddings with original embeddings through an embedding reframing design, empowering the model to generate diverse outputs. Extensive experiments demonstrate that ArtCrafter yields impressive results in visual stylization, exhibiting exceptional levels of stylistic intensity, controllability, and diversity.