SCAdapter: Content-Style Disentanglement for Diffusion Style Transfer

TL;DR

SCAdapter introduces a novel diffusion-based style transfer method that effectively disentangles content and style features using CLIP, resulting in more realistic transfers, faster inference, and improved performance over existing techniques.

Contribution

It proposes a new content-style disentanglement approach with CLIP, enabling more accurate, efficient, and controllable style transfer in diffusion models.

Findings

Outperforms state-of-the-art methods in style transfer quality.

Achieves at least 2x faster inference without optimization.

Effectively disentangles content and style for realistic results.

Abstract

Diffusion models have emerged as the leading approach for style transfer, yet they struggle with photo-realistic transfers, often producing painting-like results or missing detailed stylistic elements. Current methods inadequately address unwanted influence from original content styles and style reference content features. We introduce SCAdapter, a novel technique leveraging CLIP image space to effectively separate and integrate content and style features. Our key innovation systematically extracts pure content from content images and style elements from style references, ensuring authentic transfers. This approach is enhanced through three components: Controllable Style Adaptive Instance Normalization (CSAdaIN) for precise multi-style blending, KVS Injection for targeted style integration, and a style transfer consistency objective maintaining process coherence. Comprehensive experiments demonstrate SCAdapter significantly outperforms state-of-the-art methods in both conventional and diffusion-based baselines. By eliminating DDIM inversion and inference-stage optimization, our method achieves at least $2\times$ faster inference than other diffusion-based approaches, making it both more effective and efficient for practical applications.