GP-GAN: Towards Realistic High-Resolution Image Blending

TL;DR

This paper introduces GP-GAN, a novel high-resolution image blending framework combining classical gradient methods with GANs, achieving realistic results with fewer artifacts and outperforming existing methods.

Contribution

The paper presents the first application of GANs to high-resolution image blending, integrating Gaussian-Poisson equations with a new Blending GAN for improved quality.

Findings

Achieves state-of-the-art performance on Transient Attributes dataset.

Produces high-resolution, realistic blended images with fewer artifacts.

User study favors the proposed method over alternatives.

Abstract

It is common but challenging to address high-resolution image blending in the automatic photo editing application. In this paper, we would like to focus on solving the problem of high-resolution image blending, where the composite images are provided. We propose a framework called Gaussian-Poisson Generative Adversarial Network (GP-GAN) to leverage the strengths of the classical gradient-based approach and Generative Adversarial Networks. To the best of our knowledge, it's the first work that explores the capability of GANs in high-resolution image blending task. Concretely, we propose Gaussian-Poisson Equation to formulate the high-resolution image blending problem, which is a joint optimization constrained by the gradient and color information. Inspired by the prior works, we obtain gradient information via applying gradient filters. To generate the color information, we propose a Blending GAN to learn the mapping between the composite images and the well-blended ones. Compared to the alternative methods, our approach can deliver high-resolution, realistic images with fewer bleedings and unpleasant artifacts. Experiments confirm that our approach achieves the state-of-the-art performance on Transient Attributes dataset. A user study on Amazon Mechanical Turk finds that the majority of workers are in favor of the proposed method.

Figures

Tables

Table 1. Table 1 . Realism scores for our method and the baselines (higher is better). GP-GAN outperforms all the baselines.

Method	Input	PB(Pérez et al., 2003)	MPB(Tanaka et al., 2012)	MSB(Szeliski et al., 2011a)	Ours
Score	-0.696	-0.192	-0.151	-0.140	-0.069

Table 2. Table 2 . User study result. 4 image blending algorithms are compared on Amazon Mechanical Turk. Our method GP-GAN obtains most votes, which is consistent with the result of the realism scores.

Method	Total votes	Average votes	Std.
PB(Pérez et al., 2003)	527	1.054	1.065
MPB(Tanaka et al., 2012)	735	1.470	1.173
MSB(Szeliski et al., 2011a)	770	1.540	1.271
GP-GAN	947	1.894	1.311

Equations

$$x=x_{src}\ast x_{mask}+x_{dst}\ast (1-x_{mask}),$$

$$L(x,x_g)=\lambda L_{l_2}(x,x_g)+(1-\lambda )L_{adv}(x,x_g),$$

$$L_{l_2}(x,x_g)=\Vert G(x)-x_g{\Vert }_2^2,$$

$$L_{adv}(x,x_g)=\underset{D}{\max }{E}_{x\in \chi }[D(x_g)-D(G(x))].$$

$$H(x_h)=P(x_h)+\beta C(x_h).$$

$$P(x_h)={\int }_T\Vert \text{div}v-\Delta x_h{\Vert }_2^{2}dt,$$

$$C(x_h)={\int }_T\Vert g(x_h)-{\tilde{x}}_l{\Vert }_2^{2}dt,$$

$$v^i=\{\begin{array}{ll}\nabla x_{src}^i& \text{if }{x}_{mask}^i=1\\ \nabla x_{dst}^i& \text{if }{x}_{mask}^i=0\end{array},$$

$$H(x_h)=\Vert u-L{X}_h{\Vert }_2^2+\lambda \Vert G{X}_h-{\tilde{X}}_l{\Vert }_2^2,$$

Full text

Image Editing; Image Blending; Image Processing; Generative Adversarial Networks; Poisson Editing

††submissionid: fp435††copyright: acmlicensed††journalyear: 2019††conference: Proceedings of the 27th ACM International Conference on Multimedia; October 21–25, 2019; Nice, France††booktitle: Proceedings of the 27th ACM International Conference on Multimedia (MM ’19), October 21–25, 2019, Nice, France††price: 15.00††doi: 10.1145/3343031.3350944††isbn: 978-1-4503-6889-6/19/10††ccs: Computing methodologies Image processing††ccs: Computing methodologies Reconstruction

1. Introduction

Technologies such as PhotoShop make it much easier to edit an image than before. However, image editing still requires talents. For example, photos composited by expert users remain far better than the ones from newcomers. As the camera technologies improve, the high-resolution image makes photo editing becomes even more challenging. We want to bridge the talent gap between expert users and beginners on image editing. Mainly, we aim at addressing the problem of high-resolution image blending, which focuses on generating realistic high-resolution images given the composite ones. As shown in Figure 1, users insert an object in the background image (Figure 1(a)) and want to make it more realistic. Most users would often have high expectation on the quality of the generated images. If the algorithm produces images like Figure 1(b) or 1(c), users will give up the solution after their first few tries.

To generate well-blended images, Perez et al., Tanaka et al., and Szeliski et al. (Pérez et al., 2003; Tanaka et al., 2012; Szeliski et al., 2011a) propose the classic gradient-based methods, which enable a smooth transition and reduce the color/illumination differences between foreground and background. Among these solutions, Poisson image editing (Pérez et al., 2003) is the most widely used method, which firstly produces a gradient vector field based on the gradients of the composite image and then recovers the blended image from this gradient vector field by addressing a Poisson equation. Such methods are good at generating high-resolution results with rich details and textures. However, the generated images tend to be unrealistic, which contain various kinds of artifacts. Because the traditional gradient-based methods usually have strong assumptions about the distribution of realistic images based on human priors.

Recent researches have achieved significant progress in modeling the distribution of realistic images with the rise of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Denton et al., 2015; Radford et al., 2015; Arjovsky et al., 2017). Concretely, GANs provide a framework for estimating the distribution of natural images via simultaneously training a generator and a discriminator in a zero-sum game. The generator can produce natural images after training. Mirza et al. (Mirza and Osindero, 2014; Isola et al., 2017) generalize the idea to a condition setting, which expands the usage of GANs into image-to-image applications like image inpainting (Pathak et al., 2016). Inspired by the success of GANs in generating realistic images, we propose to employ GANs for overcoming the disadvantages of gradient-based image blending algorithms. Compared to these methods, GANs are much better at modeling the distribution of realistic images. However, it usually takes lots of computation and memory resources to generate high-resolution images with rich details and textures.

We develop a novel framework named GP-GAN to combine the strength of GANs and gradient-based image blending methods, as shown in Figure 2, which consists of two phases. In phase one, a low-resolution realistic image is generated based on the proposed Blending GAN. In phase two, we solve the proposed Gaussian-Poisson Equation based on the gradient vector field and the generated image in phase one fashioned by the Laplacian pyramid. This framework allows us to achieve high-resolution and realistic images, as shown in Figure 1(d), which outperforms all the baseline methods. Our main contributions are four folds, which are summarized as follows:

• •

We develop a framework GP-GAN for high-resolution image blending that takes advantages of both GANs and gradient-based image blending methods. To the best of our knowledge, it is the first work that explores the capability of GANs in high-resolution image blending task.

• •

We propose a network called Blending GAN for generating low-resolution realistic images.

• •

We propose the Gaussian-Poisson Equation for combining gradient information and color information.

• •

We also conduct a systematic evolution of the proposed approach based on both benchmark experiments and user studies on Amazon Mechanic Turk, which shows that our method outperforms all the baselines and achieves the state-of-the-art performance.

2. Related Work

We briefly review the relevant works from the classical image blending approaches to generative adversarial networks and conditional generative adversarial networks. We also discuss the difference between our work and the others.

2.1. Image Blending

The goal of classical image blending approaches is to improve the spatial and color consistencies between the source and target images. One way (Gracias et al., 2009) is to apply the dense image matching approach to copy and paste the corresponding pixels. However, this method would not work when there are significant differences between the source and target images. The other way is to make the transition as smooth as possible for hiding artifacts in the composite images. Alpha blending (Uyttendaele et al., 2001) is the simplest and fastest method, but it blurs the fine details when there are some registration errors between the source and target images. Alternatively, (Levin et al., 2004; Uyttendaele et al., 2001; Fattal et al., 2002; Agarwala et al., 2004; Jia et al., 2006; Kazhdan and Hoppe, 2008; Szeliski et al., 2011b) address this problem in the gradient domain. Our work is different from these gradient-based approaches in that we introduce GANs to generate a low-resolution realistic image as the color constraint, resulting in a more natural composite image. (Xue et al., 2012; Zhu et al., 2015; Tsai et al., 2017) also address a similar task to ours. However, they focus on adjusting the color and illumination of the inserted object, requiring an accurate segmentation mask. Differently, our method aims at making a smooth transition around the edges of the source and target images as well as reducing the color and illumination differences. Thus, a well-blended image can be generated by our method, although the segmentation mask of the inserted object is coarse.

2.2. Generative Adversarial Networks

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are first introduced to address the problem of generating realistic images. The main idea of GANs is a zero-sum game between learning a generator and a discriminator. The generator tries to produce more realistic images from random noises, while the discriminator aims to distinguish generated images from the real ones. Although the original method works for creating digital images from MNIST dataset, some generated images are noisy and incomprehensible. Denton et al. (Denton et al., 2015) improve the quality of the generated images by expanding GANs with a Laplacian pyramid implementation, but it does not work well for the images containing objects looking wobbly. Gregor et al. (Gregor et al., 2015) and Dosovitskiy et al. (Dosovitskiy and Brox, 2016) achieve successes in generating natural images; however, they do not leverage the generators for supervised learning. Radfor et al. (Radford et al., 2015) achieve further improvement with deeper convolutional network architecture, while Zhang et al. (Zhang et al., 2017) stack two generators to progressively render more realistic images. InfoGAN (Chen et al., 2016) learns a more interpretable latent representation. Salimans et al. (Salimans et al., 2016) reveal several tricks in training GANs. Arjovsky et al. (Arjovsky et al., 2017) introduce an alternative training method Wasserstein GAN, which relaxes the GAN training requirement of balancing the discriminator and generator. However, existing GANs still do not work well for the image editing applications in that the generated results are not high-resolution and realistic yet.

2.3. Conditional GANs

Our work is also related to conditional GANs (Mirza and Osindero, 2014), which aims to apply GANs in a conditional setting. There are several works along this research direction. Previous works apply conditional GANs to discrete labels (Mirza and Osindero, 2014), text (Reed et al., 2016), image inpainting (Pathak et al., 2016), image prediction from a normal map (Wang and Gupta, 2016), image manipulation guided by user constraints (Zhu et al., 2016), product photo generation (Yoo et al., 2016), style transfer (Li and Wand, 2016), and image-to-image translation (Isola et al., 2017). Different from previous works, we use an improved adversarial loss and discriminator for training the proposed Blending GAN. We also propose the Gaussian-Poisson Equation to produce high-resolution images.

3. The Approach

In this section, we first introduce the task of image blending formally. We then present the framework of our Gaussian-Poisson Generative Adversarial Network (GP-GAN).

3.1. Image Blending

Given a source image $x_{src}$, a destination (target) image $x_{dst}$ and a mask image $x_{mask}$, the composite (copy-and-paste) image $x$ can be obtained by Equation 1,

[TABLE]

where $\ast$ is element-wise multiplication operator. The goal of image blending is to generate a well-blended image $\tilde{x}$ that is semantically similar to the composite image $x$ but looks more realistic and natural with the resolution unchanged. $x$ is usually a high-resolution image.

3.2. Framework Overview

Generating high-resolution well-blended images is hard. To tackle this problem, we propose GP-GAN, a framework for generating high-resolution and realistic images, as shown in Figure 2. This is the first time that GANs are used for realistic high-resolution image blending to the best of our knowledge.

GP-GAN seeks a well-blended high-resolution image $\tilde{x}_{h}$ by optimizing a loss function consisting of a color constraint and a gradient constraint. The color constraint tries to make the generated image more realistic and natural while the gradient constraint captures the high-resolution details such as textures and edges.

The color constraint is constructed with a low-resolution realistic image $\tilde{x}_{l}$. To generate $\tilde{x}_{l}$, we propose Blending GAN $G(x)$ that learns to blend a copy-and-paste image and generate a realistic one semantically similar to the input. Once $G(x)$ is trained, we can use it to generate $\tilde{x}_{l}$ functioning as the color constraint.

The goal of gradient constraint is to generate the high-resolution details, including textures and edges given the composite image $x$. Their gradients directly capture textures and edges of an image. We propose Gaussian-Poisson Equation to force $\tilde{x}_{h}$ to have a similar gradient to $x$ while approximating the color of $\tilde{x}_{l}$.

GP-GAN can naturally generate realistic images in arbitrary resolution. Given a composite image $x$, we first obtain $\tilde{x}_{l}$ by feeding $x^{1}$ to $G(x)$, where $x^{1}$ is the coarsest scale in the Laplacian pyramid of $x$. Then we update $\tilde{x}_{h}^{1}$ by optimizing Gaussian-Poisson Equation with the closed-form solution. $\tilde{x}_{h}^{1}$ is upsampled and serves as $\tilde{x}_{l}$ at the finer scale in the Laplacian pyramid of $x$. The final realistic image $\tilde{x}_{h}$ with the same resolution as $x$ is obtained at the finest scale of the pyramid.

In Section 3.3, we will describe the details of our Blending GAN $G(x)$. The details of GP-GAN and Gaussian-Poisson Equation will be described in Section 3.4.

3.3. Blending GAN

We seek a low-resolution well-blended image $\tilde{x}_{l}$ that is visually realistic and semantically similar to the input image. A straightforward way is to train a conditional GAN and use the generator to produce realistic images. Since we have both the input image and the corresponding ground truth $x_{g}$, we aim to train a generator in a supervised way. To achieve this goal, we propose Blending GAN $G(x)$, which leverages the unsupervised Wasserstein GAN (Arjovsky et al., 2017) for supervised learning tasks. The proposed Blending GAN is different from Wasserstein GAN in that it has a proper constructed auxiliary loss and dedicated designed architecture.

Recent works discuss various loss functions for image processing tasks, for instance, $l_{1}$ loss (Zhao et al., 2017), $l_{2}$ loss, and perceptual loss (Johnson et al., 2016). $l_{1}$ and $l_{2}$ loss can accelerate the training process but tend to produce blurry images. The perceptual loss is good at generating high-quality images but is time and memory consuming. We employ $l_{2}$ loss in this paper because it could accelerate the training process and generate sharp and realistic images when combined with GANs (Isola et al., 2017). The combined loss function is defined as follows:

[TABLE]

where $\lambda$ is $0.999$ in our experiment. $L_{l_{2}}$ is defined as follows:

[TABLE]

and $L_{adv}$ is defined as follows:

[TABLE]

The architecture of Blending GAN $G(x)$ is shown in Figure 3, which is motivated by (Pathak et al., 2016). We find that a network with only convolutional layers could not learn to blend composite images for the lack of global information across the whole image. Thus we replace the channel-wise fully connected layer used in (Pathak et al., 2016) with standard fully connected layers.

The architecture of the discriminator $D(x)$ is shown in Figure 4. We apply the batch normalization (Ioffe and Szegedy, 2015) and leaky ReLU after each convolution except for the first layer and the last layer. The first layer employs convolution and leaky ReLU, while the last layer contains convolution only.

Training such a network needs massive data. The copy-and-paste images are easy to collect, but the ground truth images $x_{g}$ could only be obtained by expert users with image editing software, which is time-consuming. Alternatively, we use $x_{dst}$ to approximate $x_{g}$, since $x_{src}$ and $x_{dst}$ in our experiment are photos of the same scene under different conditions, e.g. season, weather, time of day, see Section 4.1 for details. Through this way, we obtain massive composite images and the corresponding ground truth, as shown in Figure 5.

3.4. Gaussian-Poisson Equation

The proposed Blending GAN can only generate low-resolution images, as shown in Figure 5. Even for slightly larger images, the results tend to be blurry with unpleasant artifacts, which is unsuitable for image blending task. Since the task usually needs to combine several high-resolution images and blend them into one realistic image with the resolution unchanged. To make use of the realistic images generated by Blending GAN, we propose Gaussian-Poisson Equation fashioned by the well-known Laplacian pyramid (Burt and Adelson, 1983) for generating high-resolution and realistic images.

We observe that although our Blending GAN cannot produce high-resolution images, the generated image $\tilde{x}_{l}$ is natural and realistic as a low-resolution image. So we can seek a high-resolution and realistic image $\tilde{x}_{h}$ by approximating the color of $\tilde{x}_{l}$ while capturing rich details like textures and edges in the original high-resolution image $x$. Such requirements are formulated into two constraints: one is the color constraint, while the other is the gradient constraint. The color constraint forces $\tilde{x}_{h}$ to have a similar color to $\tilde{x}_{l}$, which can be achieved by generating an image with the same low-frequency signals as $\tilde{x}_{l}$. The simplest way to extract the low-frequency signals is using a Gaussian filter. The gradient constraint tries to restore the high-resolution details, which is the same as forcing $\tilde{x}_{h}$ and $x$ to have the same high-frequency signals. This step could be implemented by using the divergence operator.

Formally, we need to optimize the objective function defined as follows:

[TABLE]

$P(x_{h})$ is inspired by the well-known Poisson Equation (Pérez et al., 2003) and is defined as follows:

[TABLE]

$C(x_{h})$ is defined as follows:

[TABLE]

and $\beta$ represents the color preserving parameter. We set $\beta$ to $1$ in our experiment. In Equation 6, $T$ represents the whole image region, div represents the divergence operator and $\Delta$ represents the Laplacian operator. $v$ is defined as follows:

[TABLE]

where $\nabla$ is the gradient operator. Gaussian filter is used in Equation 7 and is denoted as $g(x_{h})$. The discretized version of Equation 5 is defined as follows:

[TABLE]

where $u$ is the discretized divergence of $v$, $L$ is the matrix of the Laplacian operator, and $G$ represents the Gaussian filter. $X_{h}$ and $\tilde{X}_{l}$ are the vector representation of $x_{h}$ and $\tilde{x}_{l}$. The closed-form solution for minimizing the cost function of Equation 9 could be obtained in the same manner as (Frankot and Chellappa, 1988).

We integrate the closed-form solution for optimizing Equation 9 and the Laplacian pyramid into our final high-resolution image blending algorithm, which is described by Algorithm 1. Given a high-resolution input image $x_{src}$, $x_{dst}$ and $x_{mask}$, we first generate the low-resolution realistic image $\tilde{x}_{l}$ using Blending GAN $G(x)$. Then we generate Laplacian pyramids $x_{src}^{s},x_{dst}^{s},x_{mask}^{s},s=1,2,...,S$, where $S$ is the number of scales. $s=1$ is the coarsest scale and $s=S$ is the original resolution. We update $x_{h}^{s}$ by optimizing Equation 9 at each scale and set $\tilde{x}_{l}$ to be upsampled $x_{h}^{s}$. The final realistic image $\tilde{x}_{h}$ with the unchanged resolution is set to be $x_{h}^{S}$.

4. Experiments

In this section, the datasets for the experiments are introduced firstly. Then the training configurations and experimental settings are described. Finally, the effectiveness of our method are shown quantitatively and visually by comparing with other methods.

4.1. Dataset

Transient Attributes Database (Laffont et al., 2014) contains 8,571 images from 101 webcams. In each webcam, there are well-aligned 60-120 images with severe appearance changes caused by weather, time of day, and season, as shown in Figure 6(a) and Figure  6(b).

For training $G(x)$, we randomly select 2 images from the same camera as $x_{src}$ (Figure 6(a)) and $x_{dst}$ (Figure  6(b)). As for the ground truth $x_{g}$, we use $x_{dst}$ to approximate it since images under the same webcam is perfect-aligned. $x_{mask}$ is a binary image with a central-squared patch, as shown in Figure 6(c). The composite copy-and-paste image is then obtained by Equation 1, as shown in Figure 6(d). Although $G(x)$ is trained with the central-squared patch as the mask, our experiments show that it is still able to generate well-blended images for inputs with arbitrary masks.

To evaluate our method with arbitrary masks, we first manually annotate object-level masks for Transient Attributes Database with the LabelMe (Russell et al., 2008) annotation tool. Then we use the object-level masks to composite the copy-and-paste images, which are used to evaluate different image blending methods. The annotated mask and corresponding composite image are shown in Figure 6(e) and Figure 6(f).

4.2. Implementation Details

Our method is implemented with Chainer (Tokui et al., 2015). To train Blending GAN, we employ ADAM (Kingma and Ba, 2014) for optimization, where $\alpha$ is set to $0.002$, and $\beta_{1}$ is set to 0.5. We randomly generate 150K images from Transient Attributes Database using the central-squared patch as the mask. Then the network is trained for 25 epochs with batch size 64.

4.3. Quantitative Comparisons

Our method is compared with three classic image blending approaches. Poisson Image Editing (PB) (Pérez et al., 2003) and its improved version Modified Poisson Image Editing (MPB) (Tanaka et al., 2012) are selected as baselines because both of them employ Poisson Equation in their solutions as our method does. We also compare with multi-splines blending (MSB) (Szeliski et al., 2011a) for its effectiveness and extensive usage.

We first show the quantitative results of our method with realism score as the metric. Realism score is produced by RealismCNN (Zhu et al., 2015), which predicts the visual realism of an image regarding color, lighting, and texture compatibility.

Our method is evaluated on 500 images that are randomly sampled from Transient Attributes Database with the annotated masks. The average realism scores for our method and the baselines are shown in Table 1, where our method outperforms all the baselines. We attribute this to the nature of our method because it can learn what contributes to a realistic and natural image through adversarial learning on large datasets. The average scores are negative for all evaluated methods, which shows that many blended images are still not realistic. This suggests that there are still many improvements to be made for image blending algorithms.

4.4. User Study

Realism scores show the effectiveness of our method. Since image blending is a user-oriented task, it is essential to conduct user study for evaluation. We employ Amazon Mechanical Turk to collect user assessments. Each time, a composite image $x$ is shown to the subjects followed by three blended images produced by three different algorithms. The subjects are told to pick the most realistic image among these three blended images, as shown in Figure 7. The statistical result of user study is reported in Table 2. GP-GAN is preferred by the majority of users, which is consistent with the result of realism scores in Table 1.

4.5. Role of Blending GAN

The output of Blending GAN serves as the color constraint. In this section, we demonstrate the role of Blending GAN by replacing $\tilde{x}_{l}$ with the down-sampled composite image $x^{1}$. The blended results with either $\tilde{x}_{l}$ or $x^{1}$ as the color constraint are compared. As shown in Figure 8, the blended image tends to have more bleedings and illumination inconsistencies if $\tilde{x}_{l}$ is replaced by $x^{1}$, which shows the usefulness of low-resolution natural images in our method.

4.6. Qualitative Comparisons

Finally, we demonstrate the results of our high-resolution image blending algorithm visually by comparing with MPB and MSB. As shown in Figure 9, our method tends to generate realistic results while preserving the appearance of both $x_{src}$ and $x_{dst}$. Compared to the baseline methods, there are nearly no bleedings or illumination inconsistencies in our results while all the baseline methods have more or fewer bleedings and artifacts.

Our method can also be applied to real images in high resolution, as shown in Figure 10.

5. Conclusion

We advanced the state-of-the-art in image blending by combining the ideas from the generative model GANs and gradient-based approaches. Our insight is, on the one hand, GANs are good at generating natural images from a particular distribution but weak in capturing the high-frequency image details like textures and edges. On the other hand, the gradient-based methods perform well at generating high-resolution images with local consistency, although the generated images tend to be unnatural and have many artifacts. GANs and gradient-based methods should be integrated. Hence, this integration would result in an image blending system that overcomes the drawbacks of both approaches. Our system can also be useful for image-to-image translation task. Despite the effectiveness, our algorithm fails to generate realistic images when the composite images are far away from the distribution of the training dataset. We aim to address this issue in future work.

Acknowledgements.

This work is funded by the National Natural Science Foundation of China (Grand No. 61876181, 61721004, 61403383) and the Projects of Chinese Academy of Sciences (Grand QYZDB-SSW-JSC006 and Grand 173211KYSB20160008).

A. Visual Results

More visual results are shown in Figure 11 and Figure 12.