ZePo: Zero-Shot Portrait Stylization with Faster Sampling

TL;DR

ZePo is a fast, inversion-free portrait stylization method using diffusion models that merges content and style features in just four sampling steps, significantly improving efficiency and fidelity.

Contribution

This paper introduces ZePo, a novel diffusion-based portrait stylization framework that eliminates the need for model fine-tuning or inversion, enabling rapid stylization with high quality.

Findings

Achieves stylization in only four sampling steps

Effectively merges content and style features

Reduces computational load with feature merging

Abstract

Diffusion-based text-to-image generation models have significantly advanced the field of art content synthesis. However, current portrait stylization methods generally require either model fine-tuning based on examples or the employment of DDIM Inversion to revert images to noise space, both of which substantially decelerate the image generation process. To overcome these limitations, this paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We observed that Latent Consistency Models employing consistency distillation can effectively extract representative Consistency Features from noisy images. To blend the Consistency Features extracted from both content and style images, we introduce a Style Enhancement Attention Control technique that meticulously merges content and style features within the attention space of the target image. Moreover, we propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control. Extensive experiments have validated the effectiveness of our proposed framework in enhancing stylization efficiency and fidelity. The code is available at \url{https://github.com/liujin112/ZePo}.