Improving Style Transfer with Calibrated Metrics

TL;DR

This paper introduces a novel quantitative evaluation framework for style transfer that measures effectiveness and content coherence, revealing trade-offs and properties of various neural style transfer methods.

Contribution

It proposes calibrated metrics for style transfer evaluation and analyzes multiple methods, identifying Pareto optimal solutions and insights into style transfer performance.

Findings

Universal style transfer has high content coherence but weak style effectiveness.

Modified optimization improves both style transfer effectiveness and content preservation.

Most variability in style transfer results is due to the style itself, not the method.

Abstract

Style transfer methods produce a transferred image which is a rendering of a content image in the manner of a style image. We seek to understand how to improve style transfer. To do so requires quantitative evaluation procedures, but the current evaluation is qualitative, mostly involving user studies. We describe a novel quantitative evaluation procedure. Our procedure relies on two statistics: the Effectiveness (E) statistic measures the extent that a given style has been transferred to the target, and the Coherence (C) statistic measures the extent to which the original image's content is preserved. Our statistics are calibrated to human preference: targets with larger values of E (resp C) will reliably be preferred by human subjects in comparisons of style (resp. content). We use these statistics to investigate the relative performance of a number of Neural Style Transfer(NST) methods, revealing several intriguing properties. Admissible methods lie on a Pareto frontier (i.e. improving E reduces C or vice versa). Three methods are admissible: Universal style transfer produces very good C but weak E; modifying the optimization used for Gatys' loss produces a method with strong E and strong C; and a modified cross-layer method has slightly better E at strong cost in C. While the histogram loss improves the E statistics of Gatys' method, it does not make the method admissible. Surprisingly, style weights have relatively little effect in improving EC scores, and most variability in the transfer is explained by the style itself (meaning experimenters can be misguided by selecting styles).