In the light of feature distributions: moment matching for Neural Style Transfer

TL;DR

This paper introduces a novel moment matching approach for neural style transfer that more accurately aligns feature distributions, resulting in improved style fidelity and content disentanglement.

Contribution

It adapts the dual form of Central Moment Discrepancy to enhance feature distribution matching in neural style transfer, surpassing previous methods that only match lower-order moments.

Findings

More faithful style reproduction

Better disentanglement of style and content

Improved visual quality in style transfer

Abstract

Style transfer aims to render the content of a given image in the graphical/artistic style of another image. The fundamental concept underlying NeuralStyle Transfer (NST) is to interpret style as a distribution in the feature space of a Convolutional Neural Network, such that a desired style can be achieved by matching its feature distribution. We show that most current implementations of that concept have important theoretical and practical limitations, as they only partially align the feature distributions. We propose a novel approach that matches the distributions more precisely, thus reproducing the desired style more faithfully, while still being computationally efficient. Specifically, we adapt the dual form of Central Moment Discrepancy (CMD), as recently proposed for domain adaptation, to minimize the difference between the target style and the feature distribution of the output image. The dual interpretation of this metric explicitly matches all higher-order centralized moments and is therefore a natural extension of existing NST methods that only take into account the first and second moments. Our experiments confirm that the strong theoretical properties also translate to visually better style transfer, and better disentangle style from semantic image content.