Image Synthesis and Style Transfer

TL;DR

This paper explores synthesizing new images using deep neural network representations, specifically from the VGG network, enabling creative image transformations through inverse mapping of learned features.

Contribution

It introduces a method for image synthesis and style transfer based on inverting deep neural network representations, highlighting the potential for creative image manipulation.

Findings

Generated images demonstrate effective style transfer.

Distributed neural representations capture contours and shapes.

Inverse mapping enables novel image synthesis.

Abstract

Affine transformation, layer blending, and artistic filters are popular processes that graphic designers employ to transform pixels of an image to create a desired effect. Here, we examine various approaches that synthesize new images: pixel-based compositing models and in particular, distributed representations of deep neural network models. This paper focuses on synthesizing new images from a learned representation model obtained from the VGG network. This approach offers an interesting creative process from its distributed representation of information in hidden layers of a deep VGG network i.e., information such as contour, shape, etc. are effectively captured in hidden layers of neural networks. Conceptually, if $\Phi$ is the function that transforms input pixels into distributed representations of VGG layers ${\bf h}$, a new synthesized image $X$ can be generated from its inverse function, $X = \Phi^{-1}({\bf h})$. We describe the concept behind the approach, present some representative synthesized images and style-transferred image examples.

Equations

$$X^{\prime }\leftarrow K\ast X_i$$

$$X^{\prime }\leftarrow \alpha X_i+(1-\alpha )X_j$$

$$X\leftarrow X+\delta G_h$$

$$X\leftarrow {\Phi }^{-1}(\mathbf{h})$$

$$X\leftarrow \sum _i{w}_i{\Phi }_i^{-1}({\mathbf{h}}_i)$$

$${\mathcal{L}}_{style}=\frac{1}{4{N_l}^2{M_l}^2}\sum _{i=1}^{N_l}\sum _{j=1}^{N_l}(G_{ij}^l-A_{ij}^l{)}^2$$

$${\mathcal{L}}_{content}=\frac{1}{2}\sum _{i=1}^{N_l}\sum _{j=1}^{M_l}(F_{ij}^l-P_{ij}^l{)}^2$$

$$A_{ij}^l=\sum _{k=1}^{M_l}{S}_{ik}^l{S}_{jk}^l$$

$$G_{ij}^l=\sum _{k=1}^{M_l}{F}_{ik}^l{F}_{jk}^l$$

$$N^{r\times c}\leftarrow N^{r\times c}+\delta G_L^{r\times c}$$

$${\mathcal{L}}_c({\mathbf{h}}_n,{\mathbf{h}}_t)\propto \frac{1}{k_c}\sum _l({\mathbf{h}}_n-{\mathbf{h}}_t{)}_l^2$$

$${\mathcal{L}}_s({\mathbf{h}}_n,{\mathbf{h}}_t)\propto \frac{1}{k_s}\sum _l({\mathbf{h}}_n^T{\mathbf{h}}_n-{\mathbf{h}}_t^T{\mathbf{h}}_t{)}_l^2$$

Full text

Keywords

Image synthesis, Creative generative process, Distributed representations, VGG network

1 Introduction

Computer has been extensively employed to process 2D vector graphics and raster graphics. Human designers apply various techniques such as cropping, compositing, transforming e.g., scaling, rotating, and applying various visual effects using filtering techniques. We may further describe the graphic content processing of raster graphics into two main approaches: (i) manually or algorithmically composing existing images into a new graphical content; and (ii) creating a new graphical content using generative models that are algorithmically crafted111One of the pioneers in this area is Harold Cohen aaronshome.com/aaron/index.html [1, 2, 3, 4]; or learning from image examples, e.g., deep dream222https://github.com/google/deepdream. The first approach is the more popular contemporary techniques where graphic designers employ off-the-shelf graphics authoring tools to create new content from existing images. In this work, we are particularly interested in the second approach where a new graphical content is synthesized at pixel level using a generative model learnt from examples333In this paper, the terms synthesis and generate are often used interchangably.. This approach has received much interest in recent years.

Recent advances in deep neural networks have shed some interesting ideas on the knowledge representation and learning issues [5, 6]. It is found that hidden nodes of a deep convolution neural networks (CNNs) trained with audio or visual stimuli could represent the basic fundamental frequency of sound or basic visual patterns [7, 8]. These ideas motivate us to look into the synthesis of an image from a distributed representation. In this paper, we explore a generative model that synthesizes new images using a distributed representation obtained from VGG16 network [9]. More information about the VGG16 deep convolution neural network will be discussed in Section 4.

The rest of the paper is organized into the following sections: Section 2 discusses the background and some representative related works; Section 3 discusses our approach and gives the details of the techniques behind it; Section 4 provides a critical discussion of the output from the proposed approach; and finally, the conclusion and further research are presented in Section 5.

2 Related Works

A two dimensional image is a projection of the three dimensional world onto a 2D plane, with a primitive unit as pixel. This process abstracts the world to only pixel-intensity and colour. Features such as histograms, edges, contours, corner points, object skeleton after erosion, lines regions, Fourier coefficients, convolution filter coefficients, etc. have been extensively exploited to infer the original information of the world from pixel information. Much research in computer vision has been poured into object recognition and understanding of scenic content from features derived from pixel information. It is interesting to explore whether it would be possible to reverse the process and synthesize an image using the features mentioned.

Abstract patterns have been commonly generated using mathematical functions [10]. Random patterns, fractal and abstract patterns can normally be expressed using rather short finite length programs. Although many insights have been observed and formulated in the graphic design domain, such as the rule of third and the golden ratio, there is still a big gap in our understanding of how a non-abstract image can be automatically generated using a computer program i.e., creating an outdoor scenery on a blank canvas. Due to its complexity, researchers have often choosen to explicitly describe the generative process as a computer program [1] or describing a sequence of image processing operations to be applied to existing images, e.g., non-photo realistic rendering (NPR) [11].

In the more popular contemporary approach, a new image is created by modifying original image using transformation, artistic filters and various composition tactics. The existing images will go through various processes, where their color, shapes, texture are modified and then re-composited into a new image [12]. In this style, the content is consciously modified by a human designer who asserts all extraneous information to the composited content. Various commercial graphics software packages have been designed to assist human designers in this approach. Humans employ both top-down and bottom-up creative processes in this approach.

One of the early works attempting to automate the above creative process is from [14], the authors propose the concept named image analogies. Given images $A,A^{\prime}$ and $B$ where $f(A)\rightarrow A^{\prime}$, $f$ is a function that transform $A$ to $A^{\prime}$ e.g., $f$ could be an artistic filter function. The authors show that the transforming function $f$ could be learnt from an example pair $(A,A^{\prime})$ and then apply the learnt function to generate a new output $B^{\prime}$. This approach provides an interesting style transferred process.

With the recent advances in deep learning, it has been shown that the convolution filters of a convolution neural network (CNN) exhibit important characteristics of the way our brain responds to visual patterns; e.g., simple patterns such as lines, dots, colors that emerge in early layers and complex patterns such as textures, and compound structures that emerge in deeper layers [7, 8].

The weights of CNN can be seen as a function that re-represents an input image $I$ in hidden nodes ${\bf h}$ of CNN $f:I\mapsto{\bf h}$. Given an input image to a trained CNN, information residing in the trained CNN can be transferred to the input image by enhancing the activation signal on the hidden nodes that responds strongly to the input image. The gradient of the hidden nodes with respect to the input image could be used to modify the input image. Iteratively repeating the process will enhance the features strongly correlated with those hidden nodes e.g., abstract geometrical patterns and complex patterns that have been learnt by the network in the resulting image. Google deep dream is one of the influential works that employs this technique to generate images from its network trained with images of cats and dogs. Feeding a new input image such as the sky, the network generates a kind of hallucinative flavour by embedding parts of cats and dogs to the output images. This sparks many subsequent works in image synthesis and style transfer [15].

3 Formalizing the Pixel-based Image Synthesis Process

3.1 Composite Models

A composite model generates a new image by modifying existing pixels information through the modification of pixel intensity, through affine transformation, convolution, or compositing pixels from various sources. We highlight some important operations in the categories below.

3.1.1 Convolutional Filters:

Let $X_{i}^{r\times c}$ be a matrix of pixels of image $i$ where $x_{i}(r,c)$ is the pixel intensity. We can express convolution operation between image $i$ and a convolution kernel $K$ as

[TABLE]

where $*$ is a convolution operator and $X^{\prime}$ is the output image.

3.1.2 Layers Blending:

We can express the composition between image $i$ and $j$ as

[TABLE]

where $\alpha\in[0,1]$ controls the blending percentage of images $i$ and $j$. Various operations such as affine transformation, intensity/color manipulation, and convolution operations can be expressed as a sequence of these basic processes.

Figure 1 shows the output from composite approach. Columns one and three show the original image and columns two and four shows the processed images. The output in columns two and four are generated using Eqs. 1 and 2 respectively. The generated images closely resemble the original images. In this style, the variation is manually controlled by a human graphic designer. The process above could be automated using computers. Weaving many small generative units together into algorithms, complex processes such as drawing sunflower seeds and petals could be automated [1]. Figure 2 shows a more complex composite model generated by computer programs.

3.2 Learned Distributed Representation Models

Let $X^{r\times c}$ be a matrix of input pixels and $G_{h}^{r\times c}$ be a gradient matrix computed from hidden nodes parameters. Conceptually speaking, the generative model that optimizes the activations of hidden nodes can be expressed as:

[TABLE]

where $\delta$ is the step size, that is, an image $X$ gradually transforms according to the added gradient. Images produced in this style are interesting but the contents are random in nature since the generated image is dependent on the input pixels and the contents learnt by the network. There is no means to control the relationships among the components in the generated contents.

A generative model can be viewed as a function. If this function can be precisely determined, then, given an input image, a synthesized image can be computed and vice versa, given a synthesized image, an original image can be computed using its reverse function. Here the functionality of this generative function is emulated in an artificial neural network architecture using the process explained below.

Let $\Phi$ be the function that transforms input pixels into a feature vector h derived from hidden nodes (e.g., weights or activations) distributed in different layers of the network ${\bf h}\leftarrow\Phi(X)$. Conceptually, the original image can be reproduced from a distributed representation of ${\bf h}$ [17]:

[TABLE]

This concept can be extended to the generation of $X$ from weighted ${\bf h}_{i}$ of multiple transform functions, $i$.

[TABLE]

4 Synthesizing Images from Distributed Representations

We exploit the VGG deep neural network in our image synthesis task. VGG is the acronym for Visual Geometry Group from the University of Oxford. The group has released two fully trained deep convolution neural networks, namely VGG16 and VGG19, to the public444see www.robots.ox.ac.uk/ vgg/research/very_deep. Here, we experiment with image synthesis using parameters read from convolution layers of VGG16 network. Fig. 3 shows the architecture of VGG16 network. Each block represents a hidden layer, for example, 3$\times$3 conv, 64 denotes 64 convolution filters of size 3$\times$3.

Feeding an image $X^{r\times c}_{i}$ to the VGG network, we observe the activations in the all hidden convolution layers $l$, ${\bf h}^{l}_{i}$. In other words, activations in all hidden layers represent the pixel information of the input image and, conceptually, a copy of $X$ should be reproducible by reverse the process $X^{\prime}_{i}\leftarrow\Phi^{-1}({\bf h}^{l}_{i})$. In this work, instead of analytically solving of a function $\Phi$ and its inverse $\Phi^{-1}$, we adopt an optimization method that gradually adjusts the activation of hidden nodes to the desired ${\bf h}^{l}_{i}$.

In [18], the authors approach the style transfer task by mnimizing two kinds of loss functions: content loss and style loss. Let $P^{l}\in{\mathcal{R}}^{N_{l}\times M_{l}}$, $S^{l}\in{\mathcal{R}}^{N_{l}\times M_{l}}$ and $F^{l}\in{\mathcal{R}}^{N_{l}\times M_{l}}$ be three matrices derived from the layer $l$ of the VGG network fed with content image, style image and noise input respectively. $N_{l}$ denotes the number of feature maps and $M_{l}$ denotes the size of the feature map. The style loss and content loss are defined as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where $A_{ij}^{l}$ and $G_{ij}^{l}$ is the Gram matrix which is calculated from the inner product of $S_{ik}^{l}S_{jk}^{l}$ and $F_{ik}^{l}F_{jk}^{l}$ respectively.

4.1 Synthesized Images

Figure 4 presents twelve images in two groups. The first and the third rows are the original images; the second and the third rows are the synthesized images based on the original images. These twelve images are chosen to highlight the characteristics of synthesized images using Eq 6. Four patterns (the two-rightmost columns) are chosen to display strong long range dependency. It is clear that all synthesized images successfully capture local dependency using the ${\mathcal{L}}_{style}$ loss. However, long range dependency is not successfully captured. It is clear from the synthesized images in the two rightmost columns that although the synthesized patterns seem to capture the style (local dependency) in general, the dependency among components in a longer distance are lost and a lot of information is missing.

An image synthesized using the loss function from equation 7 displays an exact replica of the image since it is a generation based on content loss. Combining the losses from both content loss and style loss, the relationships among components can be obtained via content loss while the texture is obtained via style loss. Figure 5 shows synthesized images from equations 6 and 7 which represent content loss and style loss respectively. Combining weighted losses from both functions produces an interesting output since the pixel information from two different sources is blended together. The blending is not according to the spatial positions of the pixels(as in Eq 2) but from a deeper abstraction obtained from hidden layers of a deep neural network. This gives a kind of control known as style transfer, where one image provides information on the content and the other image provides information about the style. New synthesized images successfully capture both content and style information. This provides a new interesting generative approach.

5 Reflection & Discussion

In [14], the image analogies learn a generative function between a pair of images $g:A\mapsto A^{\prime}$ . The generative function $g(\cdot)$ learns a specific transformation which can be applied to other images. The transformation is, however, limited to the specific learnt function $g(\cdot)$. Leverage on recent advances in deep learning, pre-trained models (e.g., deep dream, VGG networks) are employed in the generative process. This allows a richer transformation style since the deep neural network acts as a transform function that re-represents information of a given image in the hidden layers. In [15], two classes of loss functions: content loss ${\mathcal{L}}_{content}$ and style loss ${\mathcal{L}}_{style}$ are proposed. This allows different combinations of style and content to be realized with ease.

We offer a summary of the creative process using distributed representation as follows: let $N^{r\times c}$ and $T^{r\times c}$ be a matrix of input pixels from white noise and the target image $T$ respectively. Feeding $N^{r\times c}$ and $T^{r\times c}$ to the VGG network produces two set of activations ${\bf h}^{l}_{n}$ and ${\bf h}^{l}_{t}$ in the hidden layers. Gradually reducing the discrepancy between ${\bf h}^{l}_{n}$ and ${\bf h}^{l}_{t}$ should conceptually synthesize the image based on information from the image $T^{r\times c}$. Let $G_{L}^{r\times c}$ be a gradient matrix computed from the loss function ${\mathcal{L}}({\bf h}_{n},{\bf h}_{i})$, hence, the generative model can be expressed as an iterative update:

[TABLE]

where $\delta$ is the step size. That is, an image $N$ gradually transforms into a new image using information from $T$. The synthesized image will share many characteristics with the original image depending on the loss functions. The content loss is, in essence, the differences between the synthesized image (initialized using white noise) and the target image:

[TABLE]

where $k_{c}$ is a constant normalizing the loss. ${\mathcal{L}}_{c}$ minimizes one to one relationship between the nodes in the hidden layers and thus preserve original content. On the other hand, the style loss ${\mathcal{L}}_{s}$ minimizes the gram matrix in the hidden layers. Minimizing the Gram matrix abstracts away spatial information (since the inner product only correlates the feature map as a whole and not the detail inside the feature map).

[TABLE]

where $k_{s}$ is a constant normalizing the loss. In [19], the authors argue that the essence of style transfer is to match the feature distribution between the style and the generated image and shows that minimizing the gram matrix is equivalent to minimizing the Maximum Mean Discrepancy (MMD) with the second order polynomial kernel.

6 Conclusion & Future Work

The synthesis of an image using the information obtained from distributed VGG layers have many strengths (i) the approach often produces visually appealing images, more appealing than those produced by a filter technique e.g., artistic filters; (ii) the approach offers a flexible means to combine different content images and style images together. The synthesized output convincingly shows that style loss produces an image with a clear local texture but often lacks a clear relationship among texture components over a long spatial distance. Source images having strong local texture such as pebble, line drawing, etc., will produce impressive outcomes.

The issue of long range dependency is a universal issue in all domains and researchers have approached this differently in different domains. For example, a Long Short-Term Memory (LSTM) [20] is an enhanced recurrent neural network that has been successfully applied to speech, text and image processing. Combining context loss and style loss to synthesize a new image offers a means to deal with long range dependency issue in images. The approach always produces interesting output since the content loss always preserves the content while the style loss decorates the existing content with the style texture. In future works, we wish to further explore how to assert controls into the generative process [1, 21].

Acknowledgments

We would like to thank the GSR office for their partial financial support given to this research.