Fast Patch-based Style Transfer of Arbitrary Style

TL;DR

This paper introduces a fast, patch-based style transfer method that combines the quality of optimization-based approaches with the efficiency of feedforward networks, enabling arbitrary style transfer with high speed.

Contribution

It proposes a new local matching objective for style transfer and trains an inverse network for real-time arbitrary style transfer, improving efficiency and flexibility.

Findings

Achieves high-quality style transfer with a simple optimization process.

Enables real-time arbitrary style transfer using a trained inverse network.

Maintains consistent results across video frames.

Abstract

Artistic style transfer is an image synthesis problem where the content of an image is reproduced with the style of another. Recent works show that a visually appealing style transfer can be achieved by using the hidden activations of a pretrained convolutional neural network. However, existing methods either apply (i) an optimization procedure that works for any style image but is very expensive, or (ii) an efficient feedforward network that only allows a limited number of trained styles. In this work we propose a simpler optimization objective based on local matching that combines the content structure and style textures in a single layer of the pretrained network. We show that our objective has desirable properties such as a simpler optimization landscape, intuitive parameter tuning, and consistent frame-by-frame performance on video. Furthermore, we use 80,000 natural images and 80,000 paintings to train an inverse network that approximates the result of the optimization. This results in a procedure for artistic style transfer that is efficient but also allows arbitrary content and style images.