Mask-guided Style Transfer Network for Purifying Real Images

TL;DR

This paper introduces a mask-guided style transfer network that purifies real outdoor images by converting them into indoor synthetic-like images, enhancing realism through region-specific feature learning and achieving state-of-the-art results.

Contribution

It proposes a novel mask-guided style transfer network with region-level loss to effectively purify real images by separating style and content features from different regions.

Findings

Effective purification of real images demonstrated on multiple datasets.

Achieved state-of-the-art performance in image style transfer.

Region-level task-guided loss improves feature separation.

Abstract

Recently, the progress of learning-by-synthesis has proposed a training model for synthetic images, which can effectively reduce the cost of human and material resources. However, due to the different distribution of synthetic images compared with real images, the desired performance cannot be achieved. To solve this problem, the previous method learned a model to improve the realism of the synthetic images. Different from the previous methods, this paper try to purify real image by extracting discriminative and robust features to convert outdoor real images to indoor synthetic images. In this paper, we first introduce the segmentation masks to construct RGB-mask pairs as inputs, then we design a mask-guided style transfer network to learn style features separately from the attention and bkgd(background) regions and learn content features from full and attention region. Moreover, we propose a novel region-level task-guided loss to restrain the features learnt from style and content. Experiments were performed using mixed studies (qualitative and quantitative) methods to demonstrate the possibility of purifying real images in complex directions. We evaluate the proposed method on various public datasets, including LPW, COCO and MPIIGaze. Experimental results show that the proposed method is effective and achieves the state-of-the-art results.

Tables

Table 1. Table 1 : Speed (in seconds) for our style transfer networks vs Gatys et al. [ 8 ] , Feifei Li et al. [ 9 ] for various resolutions. All benchmarks use a Titan X GPU.

Method	$256\times 256$	$512\times 512$	$1024\times 1024$
[1]Gatys	12.69s	45.88s	171.55s
[2]Feifei li	0.023s	0.082s	0.351s
[3]Our Method	$0.015𝐬$	$0.048𝐬$	$0.211𝐬$
[3] vs [1] speedup	$\mathrm{𝟏𝟎𝟔𝟎}𝐱$	$\mathrm{𝟏𝟎𝟐𝟔}𝐱$	$\mathrm{𝟏𝟎𝟒𝟐}𝐱$
[3] vs [2] speedup	$1.53𝐱$	$1.6𝐱$	$1.67𝐱$

Table 2. Table 2 : Test performance on MPIIGaze and purified MPIIGaze; Purified MPIIGaze is the dataset which purified by proposed method. ”Method” represents training set used with gaze estimation method. Note how purifying real dataset for training lead to improved performance.

Method	MPIIGaze	purified MPIIGaze
SVR	16.5∘	14.3∘
ALR	16.4∘	13.9∘
Random Forest(RF)	15.4∘	14.2∘
KNN with UTview	16.2∘	13.6∘
CNN with UTview	13.9∘	11.7∘
KNN with UnityEyes	12.5∘	9.9∘
CNN with UnityEyes	11.2∘	7.8∘
KNN with Syntheyes	11.4∘	8.0∘
CNN with Syntheyes	13.5∘	8.8∘

Table 3. Table 3 : Test performance on raw MPIIGaze(R MPIIGaze) and purified MPIIGaze(P MPIIGaze); Purified MPIIGaze is the dataset which purified by proposed method. ”Method” represents training set used with gaze estimation method. Note instead of refining training synthetic dataset, purifying testing real dataset is a better choice for gaze estimation.

Method	R/P MPIIGaze	error
CNN with UnityEyes	R MPIIGaze	11.2
CNN with Refined UnityEyes	R MPIIGaze	9.9
CNN with UnityEyes	P MPIIGaze	7.8

Equations

$$at{t}^{+}=\sigma (weight\ast (f_{full},f_{mask})+b)$$

$$at{t}^{+}(i,j)+at{t}^{-}(i,j)=1$$

$$\begin{array}{r}{f}_{attetion}=(f_{full},f_{mask})\otimes at{t}^{+}\\ f_{bkgd}=(f_{full},f_{mask})\otimes at{t}^{-}\end{array}$$

$${\ell }_{content}^l={\lambda }_g{\ell }_{gc}^l+{\lambda }_l{\ell }_{lc}^l$$

$${\ell }_{gc}^l=\sum _{c=1}^C\frac{1}{2N_l{M}_l}\sum _{ij}(F_{full}^{l,c}[O]-F_{full}^{l,c}[I]{)}_{ij}^2$$

$$\begin{array}{r}{\ell }_{lc}^l=\sum _{c=1}^C\frac{1}{2N_l{M}_l}\sum _{ij}(F_{full}^{l,c}[O]\ast F_{attetion}^{l,c}[I]\\ -F_{full}^{l,c}[I]\ast F_{attention}^{l,c}[I]{)}_{ij}^2\end{array}$$

$${\ell }_{style}^l={\lambda }_g{\ell }_{gs}^l+{\lambda }_l{\ell }_{ls}^l$$

$$\begin{array}{r}{\ell }_{gs}^l=\sum _{c=1}^C\frac{1}{4N_{l,c}^2{M}_{l,c}^2}\sum _{ij}{\left(G_{bkgd}^{l,c}[O]-G_{bkgd}^{l,c}[S]\right)}_{ij}^2\end{array}$$

$$\begin{array}{r}{\ell }_{ls}^l=\sum _{c=1}^C\frac{1}{4N_{l,c}^2{M}_{l,c}^2}\sum _{ij}{\left(G_{attention}^{l,c}[O]-G_{attention}^{l,c}[S]\right)}_{ij}^2\end{array}$$

$$\begin{array}{r}{G}_{bkgd}^{l,c}[O]=(F_{full}^{l,c}[O]\ast F_{bkgd}^{l,c}[I])\\ \ast (F_{full}^{l,c}[O]\ast F_{bkgd}^{l,c}[I]{)}^T\end{array}$$

$$\begin{array}{r}{G}_{bkgd}^{l,c}[S]=(F_{full}^{l,c}[S]\ast F_{bkgd}^{l,c}[S])\\ \ast (F_{full}^{l,c}[S]\ast F_{bkgd}^{l,c}[S]{)}^T\end{array}$$

$$\begin{array}{r}{G}_{attention}^{l,c}[O]=(F_{full}^{l,c}[O]\ast F_{attention}^{l,c}[I])\\ \ast (F_{full}^{l,c}[O]\ast F_{attention}^{l,c}[I]{)}^T\end{array}$$

$$\begin{array}{r}{G}_{bkgd}^{l,c}[S]=(F_{full}^{l,c}[S]\ast F_{attention}^{l,c}[S])\\ \ast (F_{full}^{l,c}[S]\ast F_{attention}^{l,c}[S]{)}^T\end{array}$$

$$L_{total}=\sum _{l=1}^L{\alpha }_l{\ell }_{content}^l+\sum _{l=1}^L{\beta }_l{\ell }_{style}^l$$

$$O=\arg \underset{I}{\min }{L}_{total}+\theta {\ell }_{TV}(I)$$

Full text

**Index Terms— ** Gaze estimation ,Style Transfer , Mask-guided Style Transfer Network , Learning-by-synthesis

1 Introduction

Recent appearance-based gaze estimation is performed under outdoor conditions by using an annotated large-scale real image training dataset. However, annotating training data sets requires a lot of manual labor. To solve this problem, a training model on a synthetic image is preferred because the annotations are automatically available. But this solution has a drawback, the distribution between the real image and the synthetic image is quite different. Traditionally, the solution is to use unmarked actual data to improve the authenticity of the synthetic image from the simulator, such as SimGANs[1], these methods only learn the global features without considering local features. In the gaze estimation task, after realization with simGANs, the shape of the pupil or the edge of the pupil might by changed , the gaze estimation error will be increased due to the wrong pupil center location. Thus, these methods cannot be applied to outdoor (field) scenes due to its weak training time and adaptability to different situations in the field. In a different manner, we try to purify real image by extracting discriminative and robust features to convert outdoor real images to indoor synthetic images. Synthetic images is more regular and easy to learn, meanwhile, the annotations are automatically available.

To avoid changing the shape of the pupil or the edge of the pupil, we propose an mask-guided style transfer network to learn both local and global features. The way to handle local features is to obtain the attention region (pupil or iris) by segmentation. Fortunately, with the rapid development of deep learning [2, 3] based image segmentation methods including FCN[4],SegNet[5],U-net[6],Mask R-CNN[7], we can obtain much better mask. To learn the style and content information from synthetic images, we first introduce the segmentation masks to construct RGB-mask pairs as inputs, then we design a mask-guided style transfer network to learn style features separately from the attention and bgkd(background) region , learn content features from full and attention region. For feature extraction, our work is most directly related to the work initiated by Gates et al.[8]. The feature map of the deep convolutional neural network with differentiated training is used to achieve the breakthrough performance of the transfer of painting style. We train a feed-forward feature extraction network for image transformation tasks. Our network aims to learn as much as possible on the premise of synthetic distribution, to minimize the loss of content transmission, and to solve the problem of insufficient spatial alignment information caused by the gram matrix. To achieve this goal, we propose a loss network with a novel task-guided loss, the attention region ,background region and full image region will be calculated in different task.

Our contributions are presented in three folds:

1. We took the first step to consider the attention region in style transfer task and propose an mask-guided style transfer network to purify the real image, making it similar to indoor conditions while retaining annotation information. Different with previous work in refining the synthetic images with global features, we purified the real images with local and global features.

2.Our network not only considers the RGB color channel, but uses the segmentation masks to construct RGB-mask pairs as inputs. We learned style features separately from the attention and bkgd(background) region and learned content features from full and attention region.

3. We proposed a hybrid research method (qualitative and quantitative) for experiments on two tasks. The results show that the proposed architecture significantly purified the real image compared with the baseline methods. Meanwhile, We achieve the state-of-the-art results on gaze estimation task.

2 Proposed Method

As shown in Figure 1, there are three multi-scale stages and a loss net to learn final features. It contains three multi-scale stages and a loss net to learn final features. There are three main streams which extracted from different regions of image , i.e. , the full-stream $f_{full}$, the attention stream $f_{attetion}$, the background stream $f_{bkgd}$. The full-stream $f_{full}$ learns features from the raw images. Meanwhile, the attention stream $f_{attetion}$ and the background stream $f_{bkgd}$ are learned attention features and background features with attention maps. The attention maps are generated by the attention subnet, the streams are designed to retain the content of input image $I_{RGB}$ and transfer the style from style image $S_{RGB}$ with the input image mask $I_{mask}$ and style image mask $S_{mask}$.

2.1 Attention Subnet

Given the input(style) image pair (RGB-Mask) as inputs, the attention subnet then produces attention maps which can be denoted as

[TABLE]

where $\sigma$ is the sigmoid function, weight and $b$ are the convolutional filter weights and bias. In the contrary, the background maps denoted as $att^{-}$ , $att^{+}$ and $att^{-}$ constitute a contrastive attention pair, for each location (i,j) which in the pair of attention maps and backgrounds maps should meet the constraint:

[TABLE]

Thus, the stream of attention and background can be denoted as :

[TABLE]

where $\otimes$ means the spatial weighting operation.

2.2 Loss network with region-level task-guided loss

With the attention maps described in last subsection, we further introduce the region-level triplet loss to enhance contrastive feature learning. After the attention operation, features from three main streams can be denoted as $f_{full}$,$f_{attetion}$ and $f_{bkgd}$, $f_{full}$,$f_{attetion}$ and $f_{bkgd}$ are used to calculate region-level task-guided loss for two tasks: keep the content and style transfer.

Our loss network can be divided into two parts: Feature reconstruction loss (a) and Style reconstruction loss(b), feature reconstruction loss is denoted as $\ell_{content}$ which is the summary of $\ell_{gc}$ and $\ell_{lc}$, meanwhile, style reconstruction loss is denoted as $\ell_{style}$ which is the summary of $\ell_{gs}$ and $\ell_{ls}$.

2.2.1 Feature reconstruction loss

Traditional feature reconstruction loss which known as content loss only takes the input image $I_{RGB}$ as input and try to minimize the loss between the content of input image $I_{RGB}$ and output image $O_{RGB}$ without considering encoding content reconstructions. We address this problem with the image segmentation masks $I_{mask}$ for the input images, the local feature of pupil region can be addressed when calculating the loss of $f_{full}$ and $f_{attetion}$. To visualise the image information that is encoded at different layers of the input image with masks, we perform gradient descent on a white noise image to find another image that matches the feature responses of the original image with mask. We then define the squared-error loss between the two feature representations

[TABLE]

[TABLE]

[TABLE]

where C is the number of channels in the semantic segmentation mask and $l$ indicates the $l$-th convolutional layer of the deep convolutional neural network, $F_{full}[\cdot]$ is the $f_{full}$ in each layer $l$ with the channel c, $F_{attention}[\cdot]$ is the $f_{attention}$ in each layer $l$ with the channel c, $\lambda_{g}$ is the weight to configure layer preferences of global losses ,$\lambda_{l}$ is the weight to configure layer preferences of local losses .

Each layer with $N_{l}$ distinct filters has $N_{l}$ feature maps each of size $M_{l}$, where $M_{l}$ is the height times the width of the feature map. So the responses in each layer $l$ can be stored in a matrix $F[\cdot]\in R^{N_{l}\times M_{l}}$ where $F[\cdot]_{ij}$ is the activation of the $i^{th}$ filter at position $j$ in each layer $l$. As minimizing $\ell_{content}$, the image content and overall spatial structure are preserved but color, texture, and exact shape are not.

2.2.2 Style reconstruction loss

Feature Gram matrices are effective at representing texture, because they capture global statistics across the image due to spatial averaging. Since textures are static, averaging over positions is required and makes Gram matrices fully blind to the global arrangement of objects inside the reference image. So if we want to keep the global arrangement of objects, make the gram matrices more controllable to compute over the exact region of entire image, we need to add some texture information to the image.

Instead of taking input image $I_{RGB}$ and style image $S_{RGB}$ as inputs, we take the input image $I_{RGB}$ and style image $S_{RGB}$ with their mask $I_{mask}$ and $S_{mask}$ as pair inputs. To learn the skin style and pupil style respectively, we denote the pupil region as attention region and extract attention maps $f_{attention}$ from both style image and input image, meanwhile, the skin region denoted as background region and product background maps $f_{bkgd}$ from style image and input image. We then define the squared-error loss between the two region feature representations

[TABLE]

[TABLE]

[TABLE]

where C is the number of channels in the semantic segmentation mask and $l$ indicates the $l$-th convolutional layer of the deep convolutional neural network. Each layer with $N_{l}$ distinct filters has $N_{l}$ feature maps each of size $M_{l}$, where $M_{l}$ is the height times the width of the feature map. So the responses in each layer $l$ can be stored in a matrix $F[\cdot]\in R^{N_{l}\times M_{l}}$ where $F[\cdot]_{ij}$ is the activation of the $i^{th}$ filter at position $j$ in each layer $l$. $G^{l,c}_{\cdot}[\cdot]$can be denoted as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where $F_{full}[\cdot]$ is the $f_{full}$ in each layer $l$ with the channel c, $F_{attention}[\cdot]$ is the $f_{attention}$ in each layer $l$ with the channel c, $F_{bkgd}[\cdot]$ is the $f_{bkgd}$ in each layer $l$ with the channel c, $\lambda_{g}$ is the weight to configure layer preferences of global losses ,$\lambda_{l}$ is the weight to configure layer preferences of local losses.

We formulate the style transfer objective by combining both two components together:

[TABLE]

where L is the total number of convolutional layers and $l$ indicates the $l$-th convolutional layer of the deep convolutional neural network. $\alpha_{l}$ and $\beta_{l}$ are the weights to configure layer preferences. $\ell_{content}$ is the content loss (Eq.(10)) and $\ell_{style}$ is the style loss(Eq.(13)). $\alpha_{l}$,$\beta_{l}$ are scalars, $\alpha_{l}=10^{2}$,$\beta_{l}=10^{4}$, in all cases the hyperparameters $\alpha_{l}$,$\beta_{l}$ are exactly the same. We find that unconstrained optimization of Equation 20 typically results in images whose pixels fall outside the range [0,255]. For a more fair comparison with our method whose output is constrained to this range, for the baseline we minimize Equation 20 using projected L-BFGS. Image O is generated by solving the problem

[TABLE]

where I is initialized with white noise. The advantage of this solution is that the requirement for mask is not too precise. It does not only retain the desired structural features, but also enhance the estimation of the pupil and iris information during the reconstruction of the style.

3 Experimental Results

3.1 Style Transfer

The purpose of the style transfer is to generate an image that combines the content of the target content image as the real image content with the style of the target style image as the style of the synthetic image. We train an image transformation network for each of the several hand selection style goals and compare our results with the baseline methods of Gatys et al.[8] and Feifei Li et al.[9]. As a baseline, we re-implemented the method of Gatys et al.[8] and Feifei Li et al.[9]. In order to make a fairer comparison with our method whose output is constrained to [0, 255], for the baseline, we minimize the equation 1 and equation 6 by using the projected L-BFGS by cropping the image to the range [0, 255] at each iteration. In most cases, the optimization converges to satisfactory results in 500 iterations.

$\mathbf{Qualitative}$ $\mathbf{Results}$: We compare the proposed style transfer method with methods proposed by Gatys et al.[8] and Feifei Li et al.[9]. Figure 4 shows the qualitative results in the indoor and outdoor scenes for the UnityEyes[10] and LPW[11] datasets respectively. From the LPW dataset, we choose six different real images to represent six different conditions of outdoor scenes, which are images (a),(b),(c),(d),(e), and (f). From the UnityEyes dataset, we selected two synthetic images that have different distributions as the style images, style A, B, none of which has no same gaze angle as the six real images. As can be seen from (a),(b), and (c), the proposed method is less affected by external factors, such as illumination, and is similar to the results obtained by Gatys et al. [8] and Feifei Li et al.[9]. However, if look closely, we can find that the proposed method is more capable of preserving the color information of the style image. From (d),(e), and (f), it is obvious that Gatys et al.[8] and Feifei Li et al.[9] are so greatly influenced by light and other factors that the pupil and the iris regions cannot be completely separated and even in (e),there is a loss of pupil area. What’s more, the distribution of pupil and iris regions is dramatically different from style image. In comparison, our proposed method can not only separate the pupil and the iris regions more easily but also the distribution of pupil and iris regions is more similar to style image.

$\mathbf{Speed}$: Table 2 compares the runtime of our method with Gatys et al.[8], Feifei Li et al.[9] for several image sizes. Across all image sizes, compared to 400 iterations of the baseline methods, our method is three orders of magnitude faster than Gatys et al.[8] and we achieve better qualitative results (Fig.6) compared with Feifei Li et al.[9] in tolerate speed. Our method processes images of size $512\times 512$ at 20 FPS, making it feasible to run in real-time or on video.

3.2 Appearance-based Gaze Estimation

$\mathbf{Implementation}$ $\mathbf{Details}$: In order to verify the effectiveness of the proposed method for gaze estimation, 3 public datasets (UTView[12], SynthesEyes[13], UnityEyes[10]) are used to train the estimator with k-NN[14][15][16][17]and CNN[18, 19, 20, 21, 22]. MPIIGaze dataset[23] and purified MPIIGaze dataset (purified by proposed method) are used for testing the accuracy.

$\mathbf{Quantitative}$ $\mathbf{Results}$: There are five gaze estimation methods used as base-line estimation methods. Three of them are common methods, Support Vector Regression(SVR), Adaptive Linear Regression(ALR) and Random Forest(RF). The other two methods are reproduced for fairly comparison with state-of-the-art. The first method is a simple cascaded method[24][25][26] which uses multiple $k$-NN($k$-Nearest Neighbor) classifier to select neighbors in feature space joint head pose, pupil center and eye appearance. The second one aims to train a simple convolutional neural network (CNN)[27][28][29] to predict the eye gaze direction by using $l_{2}$ loss. As is shown in table 2, we compare the performance of these two gaze estimation methods on different datasets, where the ”Method” represents gaze estimation methods with different training sets. It can be observed that the accuracy of gaze estimation of each dataset has improved at least three degrees, which means that our proposed method has greatly improved the performance when testing the output. This improvement will have some practical value in human-computer interaction.

Furthermore, in order to prove that purifying real dataset can achieve better performance than refining synthetic dataset on gaze estimation task, we compare proposed method with state-of-the-art methods of refining synthetic dataset for gaze estimation,SimGANs[1]. In order to make a fairer comparison with our method, we reproduce SimGANs[1], different from traditional methods which training on synthetic UnityEyes dataset and testing on real MPIIGaze dataset, SimGANs[1] refining UnityEyes with its method, try to realistic synthetic image and test on real MPIIGaze dataset. In the contrary, proposed method purify the testing real MPIIGaze dataset without modifying training synthetic datset. From Table 3, we can see that purifying testing real dataset outperforms traditional methods and SimGans[1] on gaze estimation.

$\mathbf{Preserving}$ $\mathbf{Ground}$ $\mathbf{Truth}$: We manually quantify the gaze center of the 200 real and purified images by fitting the ellipse to the pupil, which can be seen as an approximation of the direction of gaze and is difficult for humans to label accurately, to quantify that there is no significant change in the ground truth gaze direction. The absolute difference of the estimated pupil center between the real and corresponding purified images is very small: 0.8 $\pm$ 1.1 (eye width=55px)

4 Conclusion

This paper purify the real image by weakening its distribution, which is a better choice than improving the realism of synthetic image. We have applied this method to style transfer and gaze estimation tasks where we achieved comparable performance and drastically improved speed compared with existing methods. Performance evaluation indicates that purified MPIIGaze dataset (purified by our proposed method) records smaller error angle when used for gaze estimation task as compared with the raw MPIIGaze dataset.

In the future, we intend to explore modeling the real-time gaze estimation system based on the proposed method and improve the speed of purifying videos.