Semi-Supervised Learning for Face Sketch Synthesis in the Wild

TL;DR

This paper introduces a semi-supervised deep learning method for face sketch synthesis that effectively utilizes limited paired data and large unpaired face photo datasets to generate high-quality sketches in wild conditions.

Contribution

It presents a novel semi-supervised architecture that uses patch matching in feature space and pseudo sketch supervision, enabling face sketch synthesis with minimal paired data.

Findings

Achieves state-of-the-art results on public benchmarks.

Effectively handles face photos in the wild.

Utilizes small reference sets with large unpaired datasets.

Abstract

Face sketch synthesis has made great progress in the past few years. Recent methods based on deep neural networks are able to generate high quality sketches from face photos. However, due to the lack of training data (photo-sketch pairs), none of such deep learning based methods can be applied successfully to face photos in the wild. In this paper, we propose a semi-supervised deep learning architecture which extends face sketch synthesis to handle face photos in the wild by exploiting additional face photos in training. Instead of supervising the network with ground truth sketches, we first perform patch matching in feature space between the input photo and photos in a small reference set of photo-sketch pairs. We then compose a pseudo sketch feature representation using the corresponding sketch feature patches to supervise our network. With the proposed approach, we can train our networks using a small reference set of photo-sketch pairs together with a large face photo dataset without ground truth sketches. Experiments show that our method achieve state-of-the-art performance both on public benchmarks and face photos in the wild. Codes are available at https://github.com/chaofengc/Face-Sketch-Wild.

Tables

Table 1. Table 1: Details of benchmark datasets. Align: whether the sketches are well aligned with photos. Var: whether the photos have lighting variations.

Dataset		Total Pairs	Train	Test	Align	Var
CUFS	CUHK	188	88	100	✓	✗
	AR	123	80	43	✓	✗
	XM2VTS	295	100	195	✗	✗
CUFSF		1194	250	944	✗	✓

Equations

$$(i^{\prime },j^{\prime })=\underset{\begin{array}{c}{i}^{\ast }=1\sim N\\ j^{\ast }=1\sim m\end{array}}{\mathrm{argmax}}\frac{{\mathrm{\Psi }}_j\left({\mathrm{\Phi }}^l(\mathbf{x})\right)\cdot {\mathrm{\Psi }}_{j^{\ast }}\left({\mathrm{\Phi }}^l({\mathbf{x}}_{i^{\ast }}^{\mathcal{R}})\right)}{{\Vert {\mathrm{\Psi }}_j\left({\mathrm{\Phi }}^l(\mathbf{x})\right)\Vert }_2{\Vert {\mathrm{\Psi }}_{j^{\ast }}\left({\mathrm{\Phi }}^l({\mathbf{x}}_{i^{\ast }}^{\mathcal{R}})\right)\Vert }_2}.$$

$$L_p(\mathbf{x},\hat{\mathbf{y}})=\sum _{l=3}^5\sum _{j=1}^m{\Vert {\mathrm{\Psi }}_j\left({\mathrm{\Phi }}^l(\hat{\mathbf{y}})\right)-{\mathrm{\Psi }}_j^{\prime }\left({\mathrm{\Phi }}^l(\mathbf{x})\right)\Vert }_2^2,$$

$$L_{GAN\_D}=\frac{1}{2}{\mathbb{E}}_{y\sim p_{sketch}(y)}[(D(y)-1{)}^2]+\frac{1}{2}{\mathbb{E}}_{x\sim p_{photo}(x)}[(D(G(x)){)}^2]$$

$$L_{GAN\_G}={\mathbb{E}}_{x\sim p_{photo}(x)}[(D(G(x))-1{)}^2]$$

$$L_{tv}(\hat{\mathbf{y}})=\sum _{u,v}\left(({\hat{\mathbf{y}}}_{u+1,v}-{\hat{\mathbf{y}}}_{u,v}{)}^2+({\hat{\mathbf{y}}}_{u,v+1}-{\hat{\mathbf{y}}}_{u,v}{)}^2\right),$$

$$L_G={\lambda }_p{L}_p+{\lambda }_{adv}{L}_{GAN\_G}+{\lambda }_{tv}{L}_{tv},$$

$$L_D=L_{GAN\_D}$$

Full text

1 Introduction

Face sketch synthesis targets at generating a sketch from an input face photo. It has many useful applications. For instance, police officers often have to rely on face sketches to identify suspects, and face sketch synthesis makes it feasible for matching sketches against photos in a mugshot database automatically. Artists can also employ face sketch synthesis to simplify the animation production process [1]. Many people prefer using sketches as their profile pictures in social media networks [2], and face sketch synthesis allows them to produce sketches without the help of a professional artist.

Much effort has been devoted to face sketch synthesis. In particular, exemplar based methods dominated in the past two decades. These methods can achieve good performance without explicitly modeling the highly nonlinear mapping between face photos and sketches. They commonly subdivide a test photo into overlapping patches, and match these test patches with the photo patches in a reference set of photo-sketch pairs. They then compose an output sketch using the corresponding sketch patches in the reference set. Although promising results have been reported [1, 3, 4, 5], these methods have several drawbacks. For example, sketches in Fig. 4(c)(d)(e)(f) are over-smoothed and fail to preserve subtle contents such as strands of hair on the forehead. Moreover, the patch matching and optimization processes are often very time-consuming. Recent methods exploited Convolutional Neural Networks (CNNs) to learn a direct mapping between photos and sketches, which is, however, a non-trivial task. The straight forward CNN based method produces blurry sketches (see Fig. 4(g)), and methods based on Generative Adversary Networks (GAN) [6] introduce undesirable artifacts (see Fig. 4(h),(i)). Besides, all these CNN based methods do not generalize well to face photos in the wild due to the lack of large training datasets of photo-sketch pairs. Although unpaired GAN based methods such as Cycle-GAN [7] can use unpaired data to transfer images between different domains, they fail to well preserve the facial content because of the weak content constraint (see fig. 8).

In this paper, we propose a semi-supervised learning framework for face sketch synthesis that takes advantages of the exemplar based approach, the perceptual loss and GAN. We design a residual net [8] with skip connections as our generator network. Suppose we have a small reference set of photo-sketch pairs and a large face photo dataset without ground truth sketches. Similar to the exemplar based approach, we subdivide the VGG-19 [9] feature maps of the input photo into overlapping patches, and match them with the photo patches (in feature space) in the reference set. We then compose a pseudo sketch feature representation using the corresponding sketch patches (in feature space) in the reference set. We can then supervise our generator network using a perceptual loss based on the mean squared error (MSE) between the feature maps of the generated sketch and the corresponding pseudo sketch feature of the input photo. An adversary loss is also utilized to make the generated sketches more realistic.

In summary, our main contributions are three folds: (1) A semi-supervised learning framework for face sketch synthesis. Our framework allows us to train our networks using a small reference set of photo-sketch pairs together with a large face photo dataset without ground truth sketches. This enables our networks to generalize well to face photos in the wild. (2) A perceptual loss based on pseudo sketch feature. We show that the proposed loss is critical in preserving both facial content and texture details in the generated sketches. Extensive experiments are conducted to verify the effectiveness of our model. Both qualitative and quantitative results illustrate the superiority of our method. (3) To the best of our knowledge, our method is the first work that can generate visually pleasant sketches for face photos in the wild.

2 Related Works

2.1 Exemplar Based Methods

Tang and Wang [10] first introduced the exemplar based method based on eigentransformation. They projected an input photo onto the eignspace of the training photos, and then reconstruct a sketch from the eignspace of the training sketches using the same projection. Liu et al. [11] observed that the linear model holds better locally, and therefore proposed a nonlinear model based on local linear embedding (LLE). They first subdivided an input photo into overlapping patches and reconstructed each photo patch as a linear combination of the training photo patches. They then obtained the sketch patches by applying the same linear combinations to the corresponding training sketch patches. Wang and Tang [5] employed a multi-scale markov random fields (MRF) model to improve the consistency between neighboring patches. By introducing shape priors and SIFT features, Zhang et al. [12] proposed an extended version of MRF which can handle face photos under different illuminations and poses. However, these MRF based methods are not capable of synthesizing new sketch patches since they only select the best candidate sketch patch for each photo patch. To tackle this problem, Zhou et al. [3] presented the markov weight fields (MWF) model which produces a target sketch patch as a linear combination of $K$ best candidate sketch patches. Considering that patch matching based on traditional image features (e.g., PCA and SIFT) is not robust, a recent method [4] used CNN feature to represent the training patches and computed more accurate combination coefficients. To accelerate the synthesis procedure, Song et al. [1] formulated face sketch synthesis as a spatial sketch denoising (SSD) problem, and Wang et al. [13] presented an offline random sampling strategy for nearest neighbor selection of patches.

2.2 Learning Based Methods

Recent works applied CNN to synthesize sketches and produced promising results. Zhang et al. [14] proposed a 7-layer fully convolutional network (FCN) to directly transfer an input photo to a sketch. Although their model can roughly estimate the outline of a face, it fails to capture texture details with the use of intensity based mean squared error (MSE) loss. Zhang et al. [15] utilized a branched fully convolutional network (BFCN) consisting of a content branch and a texture branch. Because the face content and texture are predicted separately with different loss metrics, the final sketch looks disunited. Chen et al. [16] proposed the pyramid column feature and used it to compose a reference style for a test photo from the training sketches. They utilized a CNN to create a content image from the photo, and then transferred the reference style to introduce shadings and textures in the output sketch. Wang et al. [17] presented the multi-scale generative adversarial networks (GANs) to generate sketches from photos and vice versa. Multiple discriminators at different hidden layers are used to supervise the synthesis process. Gao et al. [18] took advantage of the facial parsing map and proposed a composition-aided stack GAN. All these deep learning based methods require ground truth photo-sketch pairs for training, and they do not generalize well to face photos in the wild due to the lack of training data.

3 Semi-Supervised Face Sketch Synthesis

3.1 Overview

Our framework is composed of three main parts, namely a generator network $G$, a pseudo sketch feature generator and a discriminator network $D$ (see Fig. 1). The generator network is a deep residual network with skip connections. It is used to generate a synthesized sketch $\hat{\mathbf{y}}$ for each input photo $\mathbf{x}$. The pseudo sketch feature generator is the key to our semi-supervised learning approach. Instead of training the generator network directly with ground truth sketches, we construct a pseudo sketch feature for each input photo to supervise the synthesis of $\hat{\mathbf{y}}$. In this way, we can train our network on any face photo datasets, and generalize our model to face photos in the wild. We further adopt a discriminator network $D$ to minimize the gap between generated sketches and real sketches drawn by artists.

3.2 Pseudo Sketch Feature Generator

Given a reference set $\mathcal{R}=\{(\mathbf{x}^{\mathcal{R}}_{i},\mathbf{y}^{\mathcal{R}}_{i})\}_{i=1}^{N}$, the pseudo sketch feature generator targets at constructing a pseudo sketch feature $\mathrm{\Phi}^{\prime}({\mathbf{x}})$ for a test photo $\mathbf{x}$ which is used to supervise the synthesis of the sketch $\hat{\mathbf{y}}$. We follow MRF-CNN [19] to extract a local patch representation of an image. We first feed $\mathbf{x}$ into a pretrained VGG-19 network and extract the feature map $\mathrm{\Phi}^{l}(\mathbf{x})$ at the $l$-th layer. Similarly, we obtain $\{\mathrm{\Phi}^{l}(\mathbf{x}^{\mathcal{R}}_{i})\}_{i=1}^{N}$ and $\{\mathrm{\Phi}^{l}(\mathbf{y}^{\mathcal{R}}_{i})\}_{i=1}^{N}$. Let us denote a $k\times k$ patch centered at a point $j$ of $\mathrm{\Phi}^{l}(\mathbf{x})$ as $\mathrm{\Psi}_{j}\left(\mathrm{\Phi}^{l}(\mathbf{x})\right)$, and the same definition applies to $\mathrm{\Psi}_{j}\left(\mathrm{\Phi}^{l}(\mathbf{x}^{\mathcal{R}}_{i})\right)$ and $\mathrm{\Psi}_{j}(\mathrm{\Phi}^{l}\left(\mathbf{y}^{\mathcal{R}}_{i})\right)$. Now for each patch $\mathrm{\Psi}_{j}\left(\mathrm{\Phi}^{l}(\mathbf{x})\right)$, where $j=1,2,\ldots,m$ and $m=(H^{l}-2\times\lfloor\frac{k}{2}\rfloor)\times(W^{l}-2\times\lfloor\frac{k}{2}\rfloor)$ with $H^{l}$ and $W^{l}$ being the height and width of $\mathrm{\Phi}^{l}(\mathbf{x})$, we find its best match $\mathrm{\Psi}_{j^{\prime}}\left(\mathrm{\Phi}^{l}(\mathbf{x}^{\mathcal{R}}_{i^{\prime}})\right)$ in the reference set based on cosine distance, i.e.,

[TABLE]

Since the photos and the corresponding sketches in $\mathcal{R}$ are well aligned, we directly apply $(i^{\prime},j^{\prime})$ to index the corresponding sketch feature patch $\mathrm{\Psi}_{j^{\prime}}\left(\mathrm{\Phi}^{l}(\mathbf{y}^{\mathcal{R}}_{i^{\prime}})\right)$ for $\mathrm{\Psi}_{j^{\prime}}\left(\mathrm{\Phi}^{l}(\mathbf{x}^{\mathcal{R}}_{i^{\prime}})\right)$, and use it as the pseudo sketch feature patch $\mathrm{\Psi}^{\prime}_{j}\left(\mathrm{\Phi}^{l}(\mathbf{x})\right)$ for $\mathrm{\Psi}_{j}\left(\mathrm{\Phi}^{l}(\mathbf{x})\right)$. Finally, a pseudo sketch feature representation (at layer $l$) for $\mathbf{x}$ is given by $\{\mathrm{\Psi}^{\prime}_{j}\left(\mathrm{\Phi}^{l}(\mathbf{x})\right)\}_{j=1}^{m}$. Fig. 2 visualizes an example of the pseudo sketch feature. It can be seen that the pseudo sketch feature provides a good approximation of the real sketch feature (see Fig. 2(a)). We also show a naïve reconstruction in Fig. 2(b) obtained by directly using the matching index to index the pixel values in the training sketches. We can see such a naïve reconstruction does roughly resemble the real sketch, which also justifies the effectiveness of the pseudo sketch feature. Note that we only need alignment between photos and sketches in $\mathcal{R}$. Since we perform a dense patch matching between the input photo and the reference photos, we can also generate reasonable pseudo sketch features for input faces under different poses (see Fig. 2(c)).

3.3 Loss Functions

Pseudo Sketch Feature Loss

We define our pseudo sketch feature loss as

[TABLE]

where $l=3,4,5$ refer to layers relu3_1, relu4_1, and relu5_1 respectively. High level features after relu3_1 are better representations of textures and more robust to appearance changes and geometric transforms [19]. Fig. 3 shows the results of using different layers in $L_{p}$. As expected, low level features (e.g., relu1_1 and relu2_1) fail to generate sketch textures. While high level features (e.g., relu5_1) can better preserve textures, they produce artifacts in terms of details (see the eyes of sketches in Fig. 3). To get better performance and reduce the computation cost of patch matching, we set $l=3,4,5$.

GAN Loss

For easier convergence, we use the least square loss when training the GAN, known as LSGAN [20]. The objective functions of LSGAN are given by

[TABLE]

Total Variation Loss

Sketches generated by CNN may be unnatural and noisy. Following previous works [21, 19, 22], we adopt the total variation loss as a natural image prior to further improve the sketch quality,

[TABLE]

where $\hat{\mathbf{y}}_{u,v}$ denotes the intensity value at $(u,v)$ of the synthesized sketch $\hat{\mathbf{y}}$.

Based on the above loss terms, we can train our generator network $G$ and discriminator network $D$ using the following two loss functions respectively:

[TABLE]

where $L_{G}$ and $L_{D}$ are minimized alternatively until convergence. $\lambda_{p}$, $\lambda_{adv}$ and $\lambda_{tv}$ are trade-off weights for each loss term respectively.

4 Implementation Details

4.1 Datasets

Photo-Sketch Pairs

We use two public datasets: the CUFS dataset(consisting of the CUHK student dataset [10], the AR dataset [23], and the XM2VTS dataset [24]) and the CUFSF dataset [25], to evaluate our model111Data comes from http://www.ihitworld.com/RSLCR.html. The CUFSF dataset is more challenging than the CUFS dataset because (1) the photos were captured under different lighting conditions and (2) the sketches exhibit strong deformation in shape and cannot be aligned with the photos well. Details of these datasets are summarized in Table 1.

Face Photos

We use the VGG-Face dataset [26] to evaluate our model on photos in the wild. There are 2,622 persons in this dataset and each person has 1,000 photos. We randomly select 2,000 persons for training and the rest for testing. For each person in the training split, we randomly select $\mathcal{N}$ photos and named the resulting dataset VGG-Face$\mathcal{N}$222The dataset will be made available., where $\mathcal{N}=01,02,\ldots,10$. We also randomly select 2 photos for each person in the testing split to construct a VGG test set of 1,244 photos.

Preprocessing

For photos/sketches which have already been aligned and have a size of $250\times 200$, we leave them unchanged. For the rest, we first detect 68 face landmarks on the image using dlib333http://dlib.net/, and calculate a similarity transform to warp the image into one with the two eyes located at $(75,125)$ and $(125,125)$ respectively. We then crop the resulting image to a size of $250\times 200$. We simply drop those photos/sketches from which we fail to detect face landmarks.

4.2 Patch Matching

As in exemplar based methods, patch matching is a time-consuming process. We accelerate this process in three ways. First, we precompute and store the feature patches for the photos and sketches in the reference set (i.e., $\{\mathrm{\Psi}_{j}\left(\mathrm{\Phi}^{l}(\mathbf{x}^{\mathcal{R}}_{i})\right)\}$ and $\{\mathrm{\Psi}_{j}(\mathrm{\Phi}^{l}\left(\mathbf{y}^{\mathcal{R}}_{i})\right)\}$). Second, instead of searching the whole reference feature set, we first identify $k$ best matched reference photos for each input photo based on the cosine distance of their relu5_1 feature maps. Patch matching is then restricted within these $k$ reference photos (we set $k=5$ in the whole training process). Third, Equ. 1 is implemented as a convolution operator which can be computed efficiently on GPU.

4.3 Training Details

We updated the generator and discriminator alternatively at every iteration. The trade-off weights $\lambda_{p}$, $\lambda_{adv}$ were set to $1$ and $10^{3}$, and $\lambda_{tv}$ was set to $10^{-5}$ when using CUFS as reference set and $10^{-2}$ when using CUFSF. We implemented our model using PyTorch444http://pytorch.org/, and trained it on a Nvidia Titan X GPU. We used Adam [27] with learning rates from $10^{-3}$ to $10^{-5}$, decreasing with a factor of $10^{-1}$. Data augmentation was done online in the color space (brightness, contrast, saturation and sharpness). Each iteration took about 2s with a batch size of 6, and the model converged after about 5 hours of training.

5 Evaluation on Public Benchmarks

In this section, we evaluate our model using two public benchmarks, namely CUFS and CUFSF, which were captured under laboratory conditions. We use the training photos from CUFS$\cup$CUFSF to train our networks. When evaluating on CUFS, the reference photo-sketch pairs only comes from CUFS, and the same applies to CUFSF. To demonstrate the effectiveness of our model, we compare our results both qualitatively and quantitatively with seven other methods, namely MWF [3], SSD [1], RSLCR [13], DGFL [4], FCN [14], Pix2Pix-GAN [28], and Cycle-GAN [7]. We also compare our results quantitatively with the latest GAN based sketch synthesis methods, i.e., PS2-MAN [29] and stack-CA-GAN [18]. Since the models of their work are not available, we can only compare with the results that are directly taken from their published papers.

5.1 Qualitative Comparison

As we can observe in Fig. 4, exemplar based methods (see Fig. 4(c),(d),(e) in general perform worse than learning based methods (see Fig. 4(g),(h),(i),(j)), especially in preserving contents of the input photos. Using deep features in exemplar based methods helps to alleviate the problem, but the results are over-smoothed (see Fig. 4(f)). Due to the lack of training data, FCN produces bad results when the photos are taken under very different lighting conditions (see last two rows of Fig. 4(g)). Although the two GANs can produce much better results than FCN, they also introduce many artifacts and noise. Thanks to the pseudo sketch feature loss, our method does not suffer from the above problems. In particular, our semi-supervised strategy allows us to incorporate more training photos without ground truth in training, which helps to improve the generalization ability.

5.2 Quantitative Comparison

5.2.1 Image Quality Assessment

For datasets with ground truth sketches (e.g., CUFS and CUFSF), previous work [13, 18, 4] typically used structural similarity (SSIM) [31] as an image quality assessment metric to measure the similarity between a generated sketch and the ground truth sketch. However, many researchers (e.g., in super resolution [32] and face sketch synthesis [30, 29]) pointed out that SSIM is not always consistent with the perceptual quality. One main reason is that SSIM favors slightly blurry images when the images contain rich textures. To demonstrate this, we show some sketches generated using different methods together with their SSIM scores in Fig. 5. It can be seen that sketch generated by RSLCR is smoother than those by Pix2Pix-GAN and our model, but have higher SSIM scores. We applied a bilateral filter to smooth all the sketches. It can be observed that the SSIM scores of the sketch generated by RSLCR remain roughly the same after smoothing, whereas those of the sketches generated by Pix2Pix-GAN and our model improve by more than $1.5\%$.

In Fig. 6(a), we show the average SSIM scores of the sketches generated by different methods on CUFS, together with the average SSIM scores of their smoothed counterparts. As expected, the average SSIM scores of most of the methods improve after smoothing, same for a few exemplar based methods which produce over-smoothed sketches. The average SSIM score of our smoothed results is comparable to that of the state-of-the-art method. In Fig. 6(b), we show the corresponding results on CUFSF. Similar conclusions can be drawn.

Due to the drawback of SSIM, we use feature similarity (FSIM) [33] as our image quality assessment metric. FSIM is better at evaluating detailed textures compared with SSIM. It can be observed from Fig. 5 that the FSIM scores of the sketches decrease after smoothing. The average FSIM scores of the sketches generated by different methods on CUFS and CUFSF are shown in Fig. 6(c) and Fig. 6(d) respectively. It can be seen that our method achieves the state-of-the-art in terms of FSIM score on both CUFS and CUFSF.

5.2.2 Face Sketch Recognition

Sketch recognition is an important application of face sketch synthesis. We follow the same practice of Wang et al. [13] and employ the null-space linear discriminant analysis (NLDA) [34] to perform the recognition experiments. Fig. 7 shows the recognition accuracy of different methods on the two datasets. Our method achieves the best result when the dimension of the reduced eigenspace is less than 100, and achieves a competitive result to the state-of-the-art method [4] when the dimension is above 100.

5.2.3 Comparison with PS2-MAN and stack-CA-GAN

To further demonstrate the effectiveness of the proposed method, we compare it with two latest GAN methods, namely PS2-MAN [29] and stack-CA-GAN [18], which are specially designed for sketch synthesis. As shown in Table 5.2.3, our method achieves the best performance on almost all datasets, except for the SSIM score in CUFSF. However, we obtain a better performance on NLDA which indicates that our model can better preserve the identify information. Note that both of these GAN methods use extra information to train their network, i.e., multi-scale supervision (PS2-MAN) and parsing map (stack-CA-GAN). Compared with them, our perceptual loss can not only avoid producing artifacts but also help to improve the generalization of the network.

6 Sketch Synthesis in the Wild

There are two challenges for sketch synthesis in the wild. The first challenge is how to deal with real photos captured under uncontrolled environments with varying pose and lighting, and cluttered backgrounds. The second is the computation time. Our method tackles the first challenge by introducing more training photos through the construction of pseudo sketch features. Regarding computation time, our CNN based model can generate a sketch in a single feed forward pass which takes about 7ms on a GPU for a $250\times 200$ photo. We compared our method with five other methods , including SSD555http://www.cs.cityu.edu.hk/~yibisong/eccv14/index.html, FCN, Pix2Pix-GAN666https://github.com/phillipi/pix2pix, Cycle-GAN777https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix, and Fast-RSLCR888http://www.ihitworld.com/RSLCR.html. In this experiment, we trained our model using CUFS as the reference set and all the training photos from the CUFS, CUFSF and VGG-Face10 as the training set. Since there are no ground truth sketches for the test photos, we carried out a mean opinion score (MOS) test to quantitatively evaluate the results.

6.1 Qualitative Comparison

As photos in the wild are captured under uncontrolled environments, their appearance may vary largely. Fig. 8 shows some photos sampled from our VGG-Face test dataset and the sketches generated by different methods. It can be observed that these photos may show very different lightings, poses, image resolutions, and hair styles. Besides, some photos may be incomplete and people may also use a cartoon as their photos for entertainment (see the last row of Fig. 8). It is therefore very difficult, if not impossible, for a method which only learns from a small set of photo-sketch pairs to generate sketches for photos in the wild. Among the results of other methods, exemplar based methods (see Fig. 8(b)(c)) fail to deal with pose changes and different hair styles. FCN produces sketches (see Fig. 8(d)) that can roughly preserve the contour of the face but lose important facial components (e.g., nose and eyes). Although GANs can generate some sketch like textures, none of them can well preserve the contents. The face shapes are distorted and the key facial parts are lost. It can be seen from Fig. 8(g) that our model can handle photos in the wild well and generate pleasant results.

6.2 Effectiveness of Additional Training Photos

Introducing more training photos from VGG-Face dataset is the key to improve the generalization ability of our model. As demonstrated in Fig. 6.2, the model trained without additional photos from VGG-Face has difficulty in handling uncontrolled lightings and different hair colors (see Fig. 6.2(b)). As we add more photos to the training set, the results improve significantly (see the eyes and hair in Fig. 6.2).

6.3 Mean Opinion Score Test

Since there are no ground truth sketches for the photos in the wild, we performed a MOS test to assess the perceptual quality of the sketches generated by different methods. Specifically, we randomly selected 30 photos from the VGG test set, and then generated the sketches for these photos using SSD, FCN, Fast-RSLCR, Pix2Pix-GAN and our method respectively. Given the example photo-sketch pairs from public benchmarks as reference, 108 raters were asked to rank 10 groups of randomly selected sketches synthesized by the five different methods. We assigned a score of 1-5 to the sketches based on their rankings (5 being the best). The results are presented in Fig. 6.2. It can be observed that the MOS of our results significantly outperforms that of the other methods. This demonstrates the superiority of our method on photos in the wild.

7 Conclusion

In this paper, we propose a semi-supervised learning framework for face sketch synthesis in the wild. We design a residual network with skip connections to transfer photos to sketches. Instead of supervising our network using ground truth sketches, we construct a novel pseudo sketch feature representation for each input photo based on feature space patch matching with a small reference set of photo-sketch pairs. This allows us to train our model using a large face photo dataset (without ground truth sketches) with the help of a small reference set of photo-sketch pairs. Training with a large face photo dataset enables our model to generalize better to photos in the wild. Experiments show that our method can produce sketches comparable to those produced by other state-of-the-art methods on four public benchmarks (in terms of SSIM and FSIM), and outperforms them on photos in the wild.

8 Acknowledgment

We thank Nannan Wang, Hao Zhou and Yibing Song for providing their codes and data. We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU used for this research.