V-Shuffle: Zero-Shot Style Transfer via Value Shuffle

TL;DR

V-Shuffle is a novel zero-shot style transfer technique that uses value shuffling within self-attention layers to balance content preservation and style fidelity, outperforming existing methods especially with a single style image.

Contribution

It introduces a new value shuffling mechanism in self-attention layers and a hybrid style regularization to improve style transfer quality.

Findings

V-Shuffle effectively reduces content leakage in style transfer.

It outperforms previous methods with a single style image.

The approach excels with multiple style images, achieving high style fidelity.

Abstract

Attention injection-based style transfer has achieved remarkable progress in recent years. However, existing methods often suffer from content leakage, where the undesired semantic content of the style image mistakenly appears in the stylized output. In this paper, we propose V-Shuffle, a zero-shot style transfer method that leverages multiple style images from the same style domain to effectively navigate the trade-off between content preservation and style fidelity. V-Shuffle implicitly disrupts the semantic content of the style images by shuffling the value features within the self-attention layers of the diffusion model, thereby preserving low-level style representations. We further introduce a Hybrid Style Regularization that complements these low-level representations with high-level style textures to enhance style fidelity. Empirical results demonstrate that V-Shuffle achieves excellent performance when utilizing multiple style images. Moreover, when applied to a single style image, V-Shuffle outperforms previous state-of-the-art methods.