Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks

TL;DR

This paper presents a semantic-aware generative approach that enhances style transfer and transforms simple doodles into detailed artworks by incorporating semantic annotations for better control and quality.

Contribution

It introduces a novel method to augment CNN-based image synthesis with semantic labels, improving control, quality, and applicability of style transfer and doodle-to-art transformations.

Findings

Improved image quality and plausibility in style transfer.

Enhanced control over generated images through semantic annotations.

Successful transformation of simple doodles into detailed artworks.

Abstract

Convolutional neural networks (CNNs) have proven highly effective at image synthesis and style transfer. For most users, however, using them as tools can be a challenging task due to their unpredictable behavior that goes against common intuitions. This paper introduces a novel concept to augment such generative architectures with semantic annotations, either by manually authoring pixel labels or using existing solutions for semantic segmentation. The result is a content-aware generative algorithm that offers meaningful control over the outcome. Thus, we increase the quality of images generated by avoiding common glitches, make the results look significantly more plausible, and extend the functional range of these algorithms---whether for portraits or landscapes, etc. Applications include semantic style transfer and turning doodles with few colors into masterful paintings!