Deep Photo Style Transfer

TL;DR

This paper presents a deep-learning method for photorealistic style transfer that maintains image content fidelity by constraining transformations to be locally affine in colorspace, effectively reducing distortions.

Contribution

It introduces a novel differentiable energy constraint to ensure locally affine color transformations, enabling photorealistic style transfer with minimal distortions.

Findings

Successfully transfers style across various scenarios including time, weather, and artistic edits.

Suppresses distortions common in previous painterly transfer methods.

Produces high-quality, photorealistic images with diverse content.

Abstract

This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style. Our approach builds upon the recent work on painterly transfer that separates style from the content of an image by considering different layers of a neural network. However, as is, this approach is not suitable for photorealistic style transfer. Even when both the input and reference images are photographs, the output still exhibits distortions reminiscent of a painting. Our contribution is to constrain the transformation from the input to the output to be locally affine in colorspace, and to express this constraint as a custom fully differentiable energy term. We show that this approach successfully suppresses distortion and yields satisfying photorealistic style transfers in a broad variety of scenarios, including transfer of the time of day, weather, season, and artistic edits.