SigStyle: Signature Style Transfer via Personalized Text-to-Image Models

TL;DR

SigStyle introduces a personalized text-to-image diffusion framework that effectively captures and transfers signature styles, including geometric patterns and color palettes, with applications in local style transfer and style-guided generation.

Contribution

The paper presents SigStyle, a novel style transfer method leveraging a hypernetwork and personalized diffusion models to explicitly capture and transfer signature styles from single images.

Findings

Outperforms existing methods in recognizing signature styles

Enables high-quality style transfer across diverse styles

Supports multiple applications like style fusion and local transfer

Abstract

Style transfer enables the seamless integration of artistic styles from a style image into a content image, resulting in visually striking and aesthetically enriched outputs. Despite numerous advances in this field, existing methods did not explicitly focus on the signature style, which represents the distinct and recognizable visual traits of the image such as geometric and structural patterns, color palettes and brush strokes etc. In this paper, we introduce SigStyle, a framework that leverages the semantic priors that embedded in a personalized text-to-image diffusion model to capture the signature style representation. This style capture process is powered by a hypernetwork that efficiently fine-tunes the diffusion model for any given single style image. Style transfer then is conceptualized as the reconstruction process of content image through learned style tokens from the personalized diffusion model. Additionally, to ensure the content consistency throughout the style transfer process, we introduce a time-aware attention swapping technique that incorporates content information from the original image into the early denoising steps of target image generation. Beyond enabling high-quality signature style transfer across a wide range of styles, SigStyle supports multiple interesting applications, such as local style transfer, texture transfer, style fusion and style-guided text-to-image generation. Quantitative and qualitative evaluations demonstrate our approach outperforms existing style transfer methods for recognizing and transferring the signature styles.