Style Transfer by Relaxed Optimal Transport and Self-Similarity

TL;DR

STROTSS introduces a novel style transfer method using relaxed optimal transport and self-similarity, enabling user control and achieving higher quality stylization compared to previous approaches.

Contribution

The paper presents a new optimization-based style transfer algorithm that incorporates user-guided control and outperforms prior methods in style-content tradeoff quality.

Findings

Higher quality stylization at any content preservation level

Effective user control over style transfer regions

Superior performance in large-scale user study

Abstract

Style transfer algorithms strive to render the content of one image using the style of another. We propose Style Transfer by Relaxed Optimal Transport and Self-Similarity (STROTSS), a new optimization-based style transfer algorithm. We extend our method to allow user-specified point-to-point or region-to-region control over visual similarity between the style image and the output. Such guidance can be used to either achieve a particular visual effect or correct errors made by unconstrained style transfer. In order to quantitatively compare our method to prior work, we conduct a large-scale user study designed to assess the style-content tradeoff across settings in style transfer algorithms. Our results indicate that for any desired level of content preservation, our method provides higher quality stylization than prior work. Code is available at https://github.com/nkolkin13/STROTSS

Tables

Table 1. Table 1: Timing comparison (in seconds) between our methods and others. The style and content images had the same dimensions and were square. *: a projected result, see text for details. -: we were not able to project these results. X: the method ran out of memory.

Image size	64	128	256	512	1024
Ours	20	38	60	95	154
Gatys	8	10	14	33	116
CNNMRF	3	8	27	117	X
Contextual	13	40	189	277	X
Reshuffle	-	-	-	69*	-

Equations

$$L(X,I_C,I_S)=\frac{\alpha {\ell }_C+{\ell }_m+{\ell }_r+\frac{1}{\alpha }{\ell }_p}{2+\alpha +\frac{1}{\alpha }}$$

$$\text{textsc}EMD(A,B)=$$

$$s.t.$$

$$\sum _i{T}_{ij}=1/n$$

$$R_A(A,B)=$$

$$R_B(A,B)=$$

$${\ell }_r=REMD(A,B)=$$

$${\ell }_r=$$

$$C_{ij}=D_{\cos }(A_i,B_j)=1-\frac{A_i\cdot B_j}{\Vert A_i\Vert \Vert B_j\Vert }$$

$${\ell }_m=\frac{1}{d}\Vert {\mu }_A-{\mu }_B{\Vert }_1+\frac{1}{d^2}\Vert {\Sigma }_A-{\Sigma }_B{\Vert }_1$$

$${\mathcal{L}}_{content}(X,C)=\frac{1}{n^2}\sum _{i,j}\left|\frac{D_{ij}^X}{{\sum }_i{D}_{ij}^X}-\frac{D_{ij}^{I_C}}{{\sum }_i{D}_{ij}^{I_C}}\right|$$

$$C_{ij}=\{\begin{array}{l}\beta \ast D_{cos}(A_i,B_j),\text{ if }i\in X_{tk},j\in S_{sk}\\ \infty ,\text{ if }\exists k\text{ s.t. }i\in X_{tk},j\notin S_{sk}\\ D_{cos}(A_i,B_j)\text{ otherwise},\end{array}$$

Full text

1 Introduction

One of the main challenges of style transfer is formalizing ’content’ and ’style’, terms which evoke strong intuitions but are hard to even define semantically. We propose formulations of each term which are novel in the domain of style transfer, but have a long history of successful application in computer vision more broadly. We hope that related efforts to refine definitions of both style and content will eventually lead to more robust recognition systems, but in this work we solely focus on their utility for style transfer.

We define style as a distribution over features extracted by a deep neural network, and measure the distance between these distributions using an efficient approximation of the Earth Movers Distance initially proposed in the Natural Language Processing community [14]. This definition of style similarity is not only well motivated statistically, but also intuitive. The goal of style transfer is to deploy the visual attributes of the style image onto the content image with minimum distortion to the content’s underlying layout and semantics; in essence to ’optimally transport’ these visual attributes.

Our definition of content is inspired by the concept of self-similarity, and the notion that human perceptual system is robust because it identifies objects based on their appearance relative to their surroundings, rather than absolute appearance. Defining content similarity in this way disconnects the term somewhat from pixels precise values making it easier to satisfy than the definitions used in prior work. This allows the output of our algorithm to maintain the perceived semantics and spatial layout of the content image, while drastically differing in pixel space.

To increase utility of style transfer as an artistic tool, it is important that users can easily and intuitively control the algorithm’s output. We extend our formulation to allow region-to-region constraints on style transfer (e.g., ensuring that hair in the content image is stylized using clouds in the style image) and point-to-point constraints (e.g., ensuring that the eye in the content image is stylized in the same way as the eye in a painting).

We quantitatively compare our method to prior work using human evaluations gathered from 662 workers on Amazon Mechanical Turk (AMT). Workers evaluated content preservation and stylization quality separately. Workers were shown two algorithms’ output for the same inputs in addition to either the content or style input, then asked which has more similar content or style respectively to the displayed input. In this way are able to quantify the performance of each algorithm along both axes. By evaluate our method and prior work for multiple hyper-parameter settings, we also measure the trade-off within each method between stylization and content preservation as hyper-parameters change. Our results indicate that for any desired level of content preservation, our method provides higher quality stylization than prior work.

2 Methods

Like the original Neural Style Transfer algorithm proposed by Gatys et al. [4] our method takes two inputs, a style image $I_{S}$ and a content image $I_{C}$, and uses the gradient descent variant RMSprop [11] to minimize our proposed objective function (equation 1) with respect to the output image $X$.

[TABLE]

We describe the content term of our loss $\alpha\ell_{C}$ in Section 2.2, and the style term $\ell_{m}+\ell_{r}+\frac{1}{\alpha}\ell_{p}$ in Section 2.3. The hyper-parameter $\alpha$ represents the relative importance of content preservation to stylization. Our method is iterative; let $X^{(t)}$ refer to the stylized output image at timestep $t$. We describe our initialization $X^{(0)}$ in Section 2.5.

2.1 Feature Extraction

Both our style and content loss terms rely upon extracting a rich feature representation from an arbitrary spatial location. In this work we use hypercolumns [21, 8] extracted from a subset of layers of VGG16 trained on ImageNet [26]. Let $\Phi(X)_{i}$ be the tensor of feature activations extracted from input image $X$ by layer $i$ of network $\Phi$. Given the set of layer indices $l_{1},..,l_{L}$ we use bilinear upsampling to match the spatial dimensions of $\Phi(X)_{l_{1}}...\Phi(X)_{l_{L}}$ to those of the original image ($X$), then concatenate all such tensors along the feature dimension. This yields a hypercolumn at each pixel, that includes features which capture low-level edge and color features, mid-level texture features, and high-level semantic features [27]. For all experiments we use all convolutional layers of VGG16 except layers 9,10,12, and 13, which we exclude because of memory constraints.

2.2 Style Loss

Let $A=\{A_{1},\ldots,A_{n}\}$ be a set of $n$ feature vectors extracted from $X^{(t)}$, and $B=\{B_{1},\ldots,B_{m}\}$ be a set of $m$ features extracted from style image $I_{S}$. The style loss is derived from the Earth Movers Distance (EMD)111Since we consider all features to have equal mass, this is a simplified version of the more general EMD [23], which allows for transport between general, non-uniform mass distributions.:

[TABLE]

where $T$ is the ’transport matrix’, which defines partial pairwise assignments, and $C$ is the ’cost matrix’ which defines how far an element in $A$ is from an element in $B$. $\textsc{EMD}(A,B)$ captures the distance between sets $A$ and $B$, but finding the optimal $T$ costs $O(\max(m,n)^{3})$, and is therefore untenable for gradient descent based style transfer (where it would need to be computed at each update step). Instead we will use the Relaxed EMD [14]. To define this we will use two auxiliary distances, essentially each is the EMD with only one of the constraints  (3) or (4):

[TABLE]

we can then define the relaxed earth movers distance as:

[TABLE]

This is equivalent to:

[TABLE]

Computing this is dominated by computing the cost matrix $C$. We compute the cost of transport (ground metric) from $A_{i}$ to $B_{j}$ as the cosince distance between the two feature vectors,

[TABLE]

We experimented with using the Euclidean distance between vectors instead, but the results were significantly worse, see the supplement for examples.

While $\ell_{r}$ does a good job of transferring the structural forms of the source image to the target, the cosine distance ignores the magnitude of the feature vectors. In practice this leads to visual artifacts in the output, most notably over-/under-saturation. To combat this we add a moment matching loss:

[TABLE]

where $\mu_{A}$, $\Sigma_{A}$ are the mean and covariance of the feature vectors in set $A$, and $\mu_{B}$ and $\Sigma_{B}$ are defined in the same way.

We also add a color matching loss, $\ell_{p}$ to encourage our output and the style image to have a similar palette. $\ell_{p}$ is defined using the Relaxed EMD between pixel colors in $X^{(t)}$ and $I_{S}$, this time and using Euclidean distance as a ground metric. We find it beneficial to convert the colors from RGB into a decorrelated colorspace with mean color as one channel when computing this term. Because palette shifting is at odds with content preservation, we weight this term by $\frac{1}{\alpha}$.

2.3 Content Loss

Our content loss is motivated by the observation that robust pattern recognition can be built using local self-similarity descriptors [25]. An every day example of this is the phenomenon called pareidolia, where the self-similarity patterns of inanimate objects are perceived as faces because they match a loose template. Formally, let $D^{X}$ be the pairwise cosine distance matrix of all (hypercolumn) feature vectors extracted from $X^{(t)}$, and let $D^{I_{C}}$ be defined analogously for the content image. We visualize several potential rows of $D^{X}$ in Figure 3. We define our content loss as:

[TABLE]

In other words the normalized cosine distance between feature vectors extracted from any pair of coordinates should remain constant between the content image and the output image. This constrains the structure of the output, without enforcing any loss directly connected to pixels of the content image. This causes the semantics and spatial layout to be broadly preserved, while allowing pixel values in $X^{(t)}$ to drastically differ from those in $I_{C}$.

2.4 User Control

We incorporate user control as constraints on the style of the output. Namely the user defines paired sets of spatial locations (regions) in $X^{(t)}$ and $I_{S}$ that must have low style loss. In the case of point-to-point user guidance each set contains only a single spatial location (defined by a click). Let us denote paired sets of spatial locations in the output and style image as $(X_{t1},S_{s1})...(X_{tK},S_{sK})$. We redefine the ground metric of the Relaxed EMD as follows:

[TABLE]

where $\beta$ controls the weight of user-specified constraints relative to the unconstrained portion of the style loss, we use $\beta=5$ in all experiments. In the case of point-to-point constraints we find it useful to augment the constraints specified by the user with 8 additional point-to-point constraints, these are automatically generated and centered around the original to form a uniform 9x9 grid. The horizontal and vertical distance between each point in the grid is set to be 20 pixels for 512x512 outputs, but this is is a tunable parameter that could be incorporated into a user interface.

2.5 Implementation Details

We apply our method iteratively at increasing resolutions, halving $\alpha$ each time. We begin with the content and style image scaled to have a long side of 64 pixels. The output at each scale is bilinearly upsampled to twice the resolution and used as initialization for the next scale. By default we stylize at four resolutions, and because we halve $\alpha$ at each resolution our default $\alpha=16.0$ is set such that $\alpha=1.0$ at the final resolution.

At the lowest resolution we initialize using the bottom level of a Laplacian pyramid constructed from the content image (high frequency gradients) added to the mean color of the style image. We then decompose the initialized output image into a five level Laplacian pyramid, and use RMSprop [11] to update entries in the pyramid to minimize our objective function. We find that optimizing the Laplacian pyramid, rather than pixels directly, dramatically speeds up convergence. At each scale we make 200 updates using RMSprop, and use a learning rate of 0.002 for all scales except the last, where we reduce it to 0.001.

The pairwise distance computation required to calculate the style and content loss precludes extracting features from all coordinates of the input images, instead we sample 1024 coordinates randomly from the style image, and 1024 coordinates in a uniform grid with a random x,y offset from the content image. We only differentiate the loss w.r.t the features extracted from these locations, and resample these locations after each step of RMSprop.

3 Related Work

Style transfer algorithms have existed for decades, and traditionally relied on hand-crafted algorithms to render an image in fixed style [7, 9], or hand-crafting features to be matched between an arbitrary style to the content image [10, 3]. The state-of-the-art was dramatically altered in 2016 when Gatys et al. [4] introduced Neural Style Transfer. This method uses features extracted from a neural network pre-trained for image classification. It defines style in terms of the Gram matrix of features extracted from multiple layers, and content as the feature tensors extracted from another set of layers. The style loss is defined as the Frobenius norm of the difference in Gram feature matrices between the output image and style image. The content loss is defined as the Frobenius norm of the difference between feature tensors from the output image and the style image. Distinct from the framework of Neural Style Transfer, there are several recent methods [17, 1] that use similarities between deep neural features to build a correspondence map between the content image and style image, and warp the style image onto the content image. These methods are extremely successful in paired settings, when the contents of the style image and content image are similar, but are not designed for style transfer between arbitrary images (unpaired or texture transfer).

Subsequent work building upon [4] has explored improvements and modifications along many axes. Perhaps the most common form of innovation is in proposals for quantifying the ’stylistic similarity’ between two images [15, 2, 22, 20]. For example in order to capture long-range spatial dependencies Berger et al. [2] propose computing multiple Gram matrices using translated feature tensors (so that the outer product is taken between feature vectors at fixed spatial offsets). Both [4] and [2] discard valuable information about the complete distribution of style features that isn’t captured by Gram matrices.

In [15] Li et al. formulate the style loss as minimizing the energy function of a Markov Random Field over the features extracted from one of the latter layers of a pretrained CNN, encouraging patches (which yielded the deep features) in the target image to match their nearest neighbor from style image in feature space. Other functionally similar losses appear in [22], which treats style transfer as matching two histograms of features, and [20], which matches features between the style and target which are significantly closer than any other pairing. In all of these methods, broadly speaking, features of the output are encouraged to lie on the support of the distribution of features extracted from the style image, but need not cover it. These losses are all similar to one component of the Relaxed EMD ($R_{A}$). However, our method differs from these approaches because our style term also encourages covering the entire distribution of features in the style image ($R_{B}$). Our style loss is most similar in spirit to that proposed by Gu et al [6], which also includes terms that encourage fidelity and diversity. Their loss minimizes the distance between explicitly paired individual patches, whereas ours minimizes the distance between distributions of features.

Another major category of innovation is replacing the optimization-based algorithm of [4] with a neural network trained to perform style transfer, enabling real-time inference. Initial efforts in this area were constrained to a limited set of pre-selected styles [13], but subsequent work relaxed this constraint and allowed arbitrary styles at test time [12]. Relative to slower optimization-based methods these works made some sacrifices in the quality of the output for speed. However, Sanakoyeu et al. [24] introduce a method for incorporating style images from the same artist into the real-time framework which produces high quality outputs in real-time, but in contrast to our work relies on having access to multiple images with the same style and requires training the style transfer mechanism separately for each new style.

Various methods have been proposed for controlling the output of style transfer. In [5] Gatys et al. propose two ’global’ control methods, that affect the entire output rather than a particular spatial region. One method is decomposing the image into hue, saturation, and luminance, and only stylizes the luminance in order to preserve the color palette of the content image. A second method from [5] is to generate an auxiliary style image either to preserve color, or to transfer style from only a particular scale (for example the transferring only the brush-strokes, rather than the larger and more structurally complex elements of the style). These types of user control are orthogonal to our method, and can be incorporated into it.

Another type of control is spatial, allowing users to ensure that certain regions of the output should be stylized using only features from a manually selected region of the style image (or that different regions of the output image should be stylized based on different style images). In [5, 18] the authors propose forms of spatial control based on the user defining matched regions of the image by creating a dense mask for both the style and content image. We demonstrate that it is straightforward to incorporate this type of user-control into our formulation of style transfer. In the supplement we show an example comparing the spatial control of our method and  [5], and demonstrate that both yield visually pleasing results that match the spatial guidance provided.

Evaluating and comparing style transfer algorithms is a challenging task because, in contrast to object recognition or segmentation, there is no established “ground truth” for the output. The most common method is a qualitative, purely subjective comparison between the output of different algorithms. Some methods also provide more refined qualitative comparisons such as texture synthesis [22, 6] and inpainting [2]. While these comparisons provide insight into the behavior of each algorithm, without quantitative comparisons it is difficult to draw conclusions about the algorithm’s performance on average. The most common quantitative evaluation is asking users to rank the output of each algorithm according to aesthetic appeal [6, 16, 19]. Recently Sanakoyeu et al. [24] propose two new forms of quantitative evaluation. The first is testing if an neural network pretrained for artist classification on real paintings can correctly classify the artist of the style image based on an algorithm’s output. The second is asking experts in art history which algorithm’s output most closely matches the style image. We designed our human evaluation study, described in section 4.1, to give a more complete sense of the trade-off each algorithm makes between content and style as its hyper-parameters vary. To the best of our knowledge it is the first such effort.

4 Experiments

We include representative qualitative results in Figures 2, 4, and an illustration of the effect of the content weight $\alpha$ in Figure 6. Figure 5 demonstrates uses of user guidance with our method.

4.1 Large-Scale Human Evaluation

Because style transfer between arbitrary content and style pairs is such a broad task, we propose three regimes that we believe cover the major use cases of style transfer. ’Paired’ refers to when the content image and style image are both representations of the same things, this is mostly images of the same category (e.g. both images of dogs), but also includes images of the same entity (e.g. both images of the London skyline). ’Unpaired’ refers to when the content and style image are not representations of the same thing (e.g. a photograph of a Central American temple, and a painting of a circus). ’Texture’ refers to when the content is a photograph of a face, and the style is a homogeneous texture (e.g. a brick wall, flames). For each regime we consider 30 style/content pairings (total of 90).

In order to quantitatively compare our method to prior work we performed several studies using AMT. An example of the workers’ interface is shown in Figure 7. Images A and B were the result of the same inputs passed into either the algorithms proposed in [4],[6], [15], [20], or our method. In Figure 7 image C is the corresponding style image, and workers were asked to choose whether the style of image is best matched by: ’A’, ’B’, ’Both Equally’, or ’Neither’. If image C is a content image, workers are posed the same question with respect to content match, instead of style. For each competing algorithm except [6] we test three sets of hyper-parameters, the defaults recommended by the authors, the same with $\frac{1}{4}$ of the content weight (high stylization), and the same with double the content weight (low stylization). Because these modifications to content weight did not alter the behavior of  [4] significantly we also tested  [4] with $\frac{1}{100}$ and $100\times$ the default content weight. We also test our method with $4\times$ the content weight. We only were able to test the default hyper-parameters for [6] because the code provided by the authors does not expose content weight as a parameter to users. We test all possible pairings of A and B between different algorithms and their hyperparameters (i.e. we do not compare an algorithm against itself with different hyperparameters, but do compare it to all hyperparameter settings of other algorithms). In each presentation, the order of output (assignment of methods to A or B in the interface) was randomized. Each pairing was voted on by an average of 4.98 different workers (minimum 4, maximum 5), 662 workers in total. On average, 3.7 workers agreed with the majority vote for each pairing. All of the images used in this evaluation will be made available to enable further benchmarking.

For an algorithm/hyper-parameter combination we define its content score to be the number of times it was selected by workers as having closer or equal content to $I_{C}$ relative to the other output it was shown with, divided by the total number of experiments it appeared in. This is always a fraction between 0 and 1. The style score is defined analogously. We present these results in Figure 8, separated by regime. The score of each point is computed over 1580 pairings on average (including the same pairings being shown to distinct workers, minimum 1410, maximum 1890). Overall for a given level of content score, our method provides a higher style score than prior work.

4.2 Ablation Study

In Figure 11 we explore the effect of different terms of our style loss, which is composed of a moment-matching loss $\ell_{m}$, the Relaxed Earth Movers Distance $\ell_{r}$, and a color palette matching loss $\ell_{p}$. As seen in Figure 11, $\ell_{m}$ alone does a decent job of transferring style, but fails to capture the larger structures of the style image. $\ell_{R_{A}}$ alone does not make use of the entire distribution of style features, and reconstructs content more poorly than $\ell_{r}$. $\ell_{R_{B}}$ alone encourages every style feature to have a nearby output feature, which is too easy to satisfy. Combining $\ell_{R_{A}}$ and $\ell_{R_{B}}$ in the relaxed earth movers distance $\ell_{r}$ results in a higher quality output than either term alone, however because the ground metric used is the cosine distance the magnitude of the features is not constrained, resulting in saturation issues. Combining $\ell_{r}$ with $\ell_{m}$ alleviates this, but some issues with the output’s palette remain, which are fixed by adding $\ell_{p}$.

4.3 Relaxed EMD Approximation Quality

To measure how well the Relaxed EMD approximates the exact Earth Movers Distance we take each of the 900 possible content/style pairings formed by the 30 content and style images used in our AMT experiments for the unpaired regime. For each pairing we compute the REMD between 1024 features extracted from random coordinates, and the exact EMD based on the same set of features. We then analyze the distribution of $\frac{REMD(A,B)}{EMD(A,B}$ Because the REMD is a lower bound, this quantity is always $\leq$1. Over the 900 image pairs, its mean was 0.60, with standard deviation 0.04. A better EMD approximation, or one that is an upper bound rather than a lower bound, may yield better style transfer results. On the other hand the REMD is simple to compute, empirically easy to optimize, and yields good results.

4.4 Timing Results

We compute our timing results using a Intel i5-7600 CPU @ 3.50GHz CPU, and a NVIDIA GTX 1080 GPU. We use square style and content images scaled to have the edge length indicated in the top row of Table 1. For inputs of size 1024x1024 the methods from [15] and [20] ran out of memory (’X’ in the table). Because the code provided by the authors [6] only runs on Windows, we had to run it on a different computer. To approximate the speed of their method on our hardware we project the timing result for 512x512 images reported in their paper based on the relative speedup for [15] between their hardware and ours. For low resolution outputs our method is relatively slow, however it scales better for outputs with resolution 512 and above relative to [15] and [20], but remains slower than [4] and our projected results for [6].

5 Conclusion and Future Work

We propose novel formalizations of style and content for style transfer and show that the resulting algorithm compares favorably to prior work, both in terms of stylization quality and content preservation. Via our ablation study we show that style-similarity losses which more accurately measure the distance between distributions of features leads to better style transfer. The approximation of the earth movers distance that we use is simple, but effective, and we leave it to future work to explore more accurate approximations. Another direction for future work is improving our method’s speed by training feed-forward style transfer methods using our proposed objective function.

6 Appendix

In order to demonstrate that our proposed method for spatial guidance gives the same level of user-control as those previously proposed we provide a qualitative comparison in Figure 10. For the same content and style with the same guidance masks we show the output of our method, and the output of the method proposed in  [5] using one of the examples from their paper.

In Figure 11 we show an extended ablation study of our method. For each content image and style image we show the effect of different style losses or algorithmic decisions on our output. ’Optimize Pixels’ refers to performing gradient descent on pixel values of the output directly, instead of the entries of a laplaccian pyramid (our default). In ’Single Scale’ we perform 800 updates at the final resolution, instead of 200 updates at each of four increasing resolutions. In ’$\ell_{2}$ Ground Metric’ we replace the ground metric of the Relaxed EMD with euclidean distance (instead of our default, cosine distance). The other style loss ablations are explained in Section 4.2 of the main text.