DiffArtist: Towards Structure and Appearance Controllable Image Stylization

TL;DR

DiffArtist introduces a novel 2D image stylization method that independently controls both structural and appearance styles, enabling more precise and flexible artistic transformations without additional tuning.

Contribution

It is the first to provide simultaneous, fine-grained control over structure and appearance in neural stylization using separate diffusion processes.

Findings

Achieves superior style fidelity and dual-controllability.

Outperforms existing methods in style transfer quality.

Uses a training-free, text-driven approach.

Abstract

Artistic styles are defined by both their structural and appearance elements. Existing neural stylization techniques primarily focus on transferring appearance-level features such as color and texture, often neglecting the equally crucial aspect of structural stylization. To address this gap, we introduce \textbf{DiffArtist}, the first 2D stylization method to offer fine-grained, simultaneous control over both structure and appearance style strength. This dual controllability is achieved by representing structure and appearance generation as separate diffusion processes, necessitating no further tuning or additional adapters. To properly evaluate this new capability of dual stylization, we further propose a Multimodal LLM-based stylization evaluator that aligns significantly better with human preferences than existing metrics. Extensive analysis shows that DiffArtist achieves superior style fidelity and dual-controllability compared to state-of-the-art methods. Its text-driven, training-free design and unprecedented dual controllability make it a powerful and interactive tool for various creative applications. Project homepage: https://diffusionartist.github.io.