MOSAIC: Multi-Object Segmented Arbitrary Stylization Using CLIP

TL;DR

MOSAIC introduces a novel text-guided method for object-wise image stylization, enabling fine control over individual objects' styles based on context, surpassing previous methods in quality and flexibility.

Contribution

It is the first approach to achieve text-guided, arbitrary object-wise stylization using vision transformer-based segmentation and stylization modules.

Findings

Produces high-quality, visually appealing stylized images.

Enhances control over stylization of individual objects.

Generalizes well to unseen object classes.

Abstract

Style transfer driven by text prompts paved a new path for creatively stylizing the images without collecting an actual style image. Despite having promising results, with text-driven stylization, the user has no control over the stylization. If a user wants to create an artistic image, the user requires fine control over the stylization of various entities individually in the content image, which is not addressed by the current state-of-the-art approaches. On the other hand, diffusion style transfer methods also suffer from the same issue because the regional stylization control over the stylized output is ineffective. To address this problem, We propose a new method Multi-Object Segmented Arbitrary Stylization Using CLIP (MOSAIC), that can apply styles to different objects in the image based on the context extracted from the input prompt. Text-based segmentation and stylization modules which are based on vision transformer architecture, were used to segment and stylize the objects. Our method can extend to any arbitrary objects, styles and produce high-quality images compared to the current state of art methods. To our knowledge, this is the first attempt to perform text-guided arbitrary object-wise stylization. We demonstrate the effectiveness of our approach through qualitative and quantitative analysis, showing that it can generate visually appealing stylized images with enhanced control over stylization and the ability to generalize to unseen object classes.