StyleTokenizer: Defining Image Style by a Single Instance for Controlling Diffusion Models

TL;DR

StyleTokenizer introduces a novel zero-shot method for controlling image style in diffusion models by aligning style representations with text, using a new dataset and a style tokenizer to accurately capture style from a single image.

Contribution

The paper presents StyleTokenizer, a new approach that aligns style and text representations, enabling effective style control from a single image without compromising text prompt effectiveness.

Findings

Accurately captures style from a single reference image.

Generates images consistent with style and text prompts.

Outperforms existing methods in style control quality.

Abstract

Despite the burst of innovative methods for controlling the diffusion process, effectively controlling image styles in text-to-image generation remains a challenging task. Many adapter-based methods impose image representation conditions on the denoising process to accomplish image control. However these conditions are not aligned with the word embedding space, leading to interference between image and text control conditions and the potential loss of semantic information from the text prompt. Addressing this issue involves two key challenges. Firstly, how to inject the style representation without compromising the effectiveness of text representation in control. Secondly, how to obtain the accurate style representation from a single reference image. To tackle these challenges, we introduce StyleTokenizer, a zero-shot style control image generation method that aligns style representation with text representation using a style tokenizer. This alignment effectively minimizes the impact on the effectiveness of text prompts. Furthermore, we collect a well-labeled style dataset named Style30k to train a style feature extractor capable of accurately representing style while excluding other content information. Experimental results demonstrate that our method fully grasps the style characteristics of the reference image, generating appealing images that are consistent with both the target image style and text prompt. The code and dataset are available at https://github.com/alipay/style-tokenizer.