Sem-CS: Semantic CLIPStyler for Text-Based Image Style Transfer

TL;DR

Sem-CS advances text-based image style transfer by incorporating semantic segmentation, ensuring style is applied appropriately to different image regions, resulting in more realistic and semantically consistent stylized images.

Contribution

We introduce Semantic CLIPStyler (Sem-CS), a novel method that performs semantic-aware style transfer using text descriptions and image segmentation, improving over prior style spill-over issues.

Findings

Superior qualitative results in style transfer quality

Quantitative improvements in DISTS and NIMA scores

Positive user study feedback

Abstract

CLIPStyler demonstrated image style transfer with realistic textures using only a style text description (instead of requiring a reference style image). However, the ground semantics of objects in the style transfer output is lost due to style spill-over on salient and background objects (content mismatch) or over-stylization. To solve this, we propose Semantic CLIPStyler (Sem-CS), that performs semantic style transfer. Sem-CS first segments the content image into salient and non-salient objects and then transfers artistic style based on a given style text description. The semantic style transfer is achieved using global foreground loss (for salient objects) and global background loss (for non-salient objects). Our empirical results, including DISTS, NIMA and user study scores, show that our proposed framework yields superior qualitative and quantitative performance. Our code is available at github.com/chandagrover/sem-cs.