AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation

TL;DR

AI Illustrator introduces a prompt-based cross-modal framework leveraging pre-trained models to translate complex textual descriptions into visually appealing images, enhancing automatic book illustration with semantic accuracy and style adaptation.

Contribution

This work presents a novel framework combining CLIP and StyleGAN for translating raw descriptions into images without external paired data, and introduces a new benchmark for evaluation.

Findings

Outperforms existing methods in handling complex descriptions

Successfully generates semantically consistent images

Provides a new benchmark with 200 descriptions

Abstract

AI illustrator aims to automatically design visually appealing images for books to provoke rich thoughts and emotions. To achieve this goal, we propose a framework for translating raw descriptions with complex semantics into semantically corresponding images. The main challenge lies in the complexity of the semantics of raw descriptions, which may be hard to be visualized (e.g., "gloomy" or "Asian"). It usually poses challenges for existing methods to handle such descriptions. To address this issue, we propose a Prompt-based Cross-Modal Generation Framework (PCM-Frame) to leverage two powerful pre-trained models, including CLIP and StyleGAN. Our framework consists of two components: a projection module from Text Embeddings to Image Embeddings based on prompts, and an adapted image generation module built on StyleGAN which takes Image Embeddings as inputs and is trained by combined semantic consistency losses. To bridge the gap between realistic images and illustration designs, we further adopt a stylization model as post-processing in our framework for better visual effects. Benefiting from the pre-trained models, our method can handle complex descriptions and does not require external paired data for training. Furthermore, we have built a benchmark that consists of 200 raw descriptions. We conduct a user study to demonstrate our superiority over the competing methods with complicated texts. We release our code at https://github.com/researchmm/AI_Illustrator.