Paint it Black: Generating paintings from text descriptions

TL;DR

This paper explores methods for generating artistic paintings from text descriptions by combining photorealistic image generation with style transfer, addressing a less-studied intersection of text-to-image synthesis and style transfer.

Contribution

It introduces two integrated strategies for generating paintings from captions, combining photorealistic image synthesis with style transfer and fine-tuning on captioned paintings.

Findings

Models are evaluated with various metrics and user studies.

The approaches demonstrate promising results in generating artistic images from text.

The paper advances the understanding of combining caption-based image generation with style transfer.

Abstract

Two distinct tasks - generating photorealistic pictures from given text prompts and transferring the style of a painting to a real image to make it appear as though it were done by an artist, have been addressed many times, and several approaches have been proposed to accomplish them. However, the intersection of these two, i.e., generating paintings from a given caption, is a relatively unexplored area with little data available. In this paper, we have explored two distinct strategies and have integrated them together. First strategy is to generate photorealistic images and then apply style transfer and the second strategy is to train an image generation model on real images with captions and then fine-tune it on captioned paintings later. These two models are evaluated using different metrics as well as a user study is conducted to get human feedback on the produced results.