Joint Bilateral Learning for Real-time Universal Photorealistic Style Transfer

TL;DR

This paper introduces a fast, end-to-end neural network for photorealistic style transfer that produces high-quality, artifact-free results in real-time, even at 4K resolution on mobile devices.

Contribution

The authors present a novel local edge-aware affine transform approach within a neural network for photorealistic style transfer, achieving real-time performance and superior visual quality.

Findings

Produces visually superior results compared to state-of-the-art methods.

Achieves three orders of magnitude faster processing, enabling real-time 4K performance.

Validated through ablation and user studies.

Abstract

Photorealistic style transfer is the task of transferring the artistic style of an image onto a content target, producing a result that is plausibly taken with a camera. Recent approaches, based on deep neural networks, produce impressive results but are either too slow to run at practical resolutions, or still contain objectionable artifacts. We propose a new end-to-end model for photorealistic style transfer that is both fast and inherently generates photorealistic results. The core of our approach is a feed-forward neural network that learns local edge-aware affine transforms that automatically obey the photorealism constraint. When trained on a diverse set of images and a variety of styles, our model can robustly apply style transfer to an arbitrary pair of input images. Compared to the state of the art, our method produces visually superior results and is three orders of magnitude faster, enabling real-time performance at 4K on a mobile phone. We validate our method with ablation and user studies.