Bridging Text and Image for Artist Style Transfer via Contrastive Learning

TL;DR

This paper introduces CLAST, a novel method for artistic style transfer that uses contrastive learning and text-image encoders to enable flexible, text-driven style transfer without online fine-tuning, achieving high-quality results efficiently.

Contribution

The paper presents a new contrastive learning approach leveraging CLIP for text-driven style transfer, introducing adaLN-based style-content fusion, and demonstrating superior performance over existing methods.

Findings

Outperforms state-of-the-art style transfer methods

Operates without online fine-tuning

Processes 512x512 images in 0.03 seconds

Abstract

Image style transfer has attracted widespread attention in the past few years. Despite its remarkable results, it requires additional style images available as references, making it less flexible and inconvenient. Using text is the most natural way to describe the style. More importantly, text can describe implicit abstract styles, like styles of specific artists or art movements. In this paper, we propose a Contrastive Learning for Artistic Style Transfer (CLAST) that leverages advanced image-text encoders to control arbitrary style transfer. We introduce a supervised contrastive training strategy to effectively extract style descriptions from the image-text model (i.e., CLIP), which aligns stylization with the text description. To this end, we also propose a novel and efficient adaLN based state space models that explore style-content fusion. Finally, we achieve a text-driven image style transfer. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods in artistic style transfer. More importantly, it does not require online fine-tuning and can render a 512x512 image in 0.03s.