Exploring the structure of a real-time, arbitrary neural artistic stylization network

TL;DR

This paper introduces a real-time neural style transfer network that combines flexibility and speed, capable of stylizing any content with any style image, trained on a large painting dataset.

Contribution

It presents a novel method that predicts normalization parameters directly from style images, enabling flexible, real-time artistic stylization with an unsupervised learned embedding space.

Findings

Successfully trained on 80,000 paintings

Generalizes to unseen paintings

Embedding space is smooth and semantically meaningful

Abstract

In this paper, we present a method which combines the flexibility of the neural algorithm of artistic style with the speed of fast style transfer networks to allow real-time stylization using any content/style image pair. We build upon recent work leveraging conditional instance normalization for multi-style transfer networks by learning to predict the conditional instance normalization parameters directly from a style image. The model is successfully trained on a corpus of roughly 80,000 paintings and is able to generalize to paintings previously unobserved. We demonstrate that the learned embedding space is smooth and contains a rich structure and organizes semantic information associated with paintings in an entirely unsupervised manner.