GAN Based Image Deblurring Using Dark Channel Prior

TL;DR

This paper introduces a GAN-based image deblurring method that incorporates dark channel prior into the loss function, resulting in improved performance, robustness to noise, and a lightweight network architecture.

Contribution

It presents a novel GAN framework for image deblurring that integrates dark channel prior into the loss function, replacing the indifferentiable form with an L2 norm for compatibility.

Findings

Improved deblurring quality on synthetic and real images

Enhanced robustness to image noise

Lighter network structure reduces training and testing time

Abstract

A conditional general adversarial network (GAN) is proposed for image deblurring problem. It is tailored for image deblurring instead of just applying GAN on the deblurring problem. Motivated by that, dark channel prior is carefully picked to be incorporated into the loss function for network training. To make it more compatible with neuron networks, its original indifferentiable form is discarded and L2 norm is adopted instead. On both synthetic datasets and noisy natural images, the proposed network shows improved deblurring performance and robustness to image noise qualitatively and quantitatively. Additionally, compared to the existing end-to-end deblurring networks, our network structure is light-weight, which ensures less training and testing time.

Figures

Tables

Table 1. Table 1: Table. 1 Average PSNR and SSIM.

Dataset	Metrics	[9]	$dc0$	$dc250$	$dc250p$
Original	PSNR	26.63	26.70	27.01	26.45
	SSIM	0.8701	0.8798	0.8813	0.8680
Noisy	PSNR	26.32	26.53	26.83	26.31
	SSIM	0.8524	0.8697	0.8707	0.8604

Equations

$$\underset{G}{\min }\underset{D}{\max }{\mathbb{E}}_{\bar{x}\sim {\text{P}}_{\text{data}}(\bar{x})}\left[\log D(\bar{x})\right]+{\mathbb{E}}_{z\sim {\text{P}}_z(z)}\left[\log (1-D(G(z)))\right],$$

$$\underset{G}{\min }\underset{D}{\max }{\mathbb{E}}_{\bar{x}\sim {\text{P}}_{\text{data}}(\bar{x})}\left[\log D(\bar{x},y)\right]+{\mathbb{E}}_{z\sim {\text{P}}_z(z)}\left[\log (1-D(G(z,y),y))\right],$$

$${\mathfrak{D}}_c(p)=\underset{q\in \mathcal{N}(p)}{\min }\left(\underset{c\in \{r,g,b\}}{\min }{I}^c(q)\right),$$

$${\mathcal{L}}_d={\mathbb{E}}_{\bar{x},y}\left[\log D(\bar{x},y)\right]+{\mathbb{E}}_{y,z}\left[\log (1-D(G(z,y),y))\right].$$

$${\mathcal{L}}_g={\mathbb{E}}_{y,z}\left[\log (1-D(G(z,y),y))\right]+{\lambda }_1\cdot {\mathcal{L}}_{content}+{\lambda }_2\cdot {\mathcal{L}}_{darkchannel},$$

$${\mathcal{L}}_{content}={\mathbb{E}}_{\bar{x},y,z}\left[||\bar{x}-G(y,z)|{|}_1\right].$$

$${\mathcal{L}}_{darkchannel}={\mathbb{E}}_{\bar{x},y,z}\left[||{\mathfrak{D}}_c(\bar{x})-{\mathfrak{D}}_c(G(y,z))|{|}_2\right].$$

Full text

Introduction

Blur is a common artifact for images taken by hand-held cameras. It is mostly caused by object motions, hand shake or out-of-focus. The blurry image is often modeled as convolution of a sharp image and a blur kernel. And the target of deblurring is to restore a latent sharp image from the blurry one. Single image deblurring, however, is a highly ill-posed problem, since it contains insufficient information to recover a unique sharp image.

In the past few years, assorted constraints and regulation schemes have been proposed to exclude implausible solutions. Priors, like total variation prior [1], sparse image prior [2], heavy-tailed gradient prior [3] and dark channel prior [4], are combined with $L_{1}/L_{2}$ norm image regulation term to suppress ringing artifacts and improve quality. Zhen [5] takes advantage of inertia sensor data to gain extra information and estimate spatially varying blur kernels. However, since the blur kernel in reality is more complicated than the model, estimation of blur kernel is inaccurate, which causes ringing artifacts. Furthermore, these methods based on iterative optimization techniques are computationally intensive.

Recently, Convolutional Neural Networks (CNN) and deep learning related techniques have drawn a great attention in computer vision and image processing. Their applications in image deblurring demonstrate promising results. Sun [7] and Schuler [6] use CNN to estimate the spatially-invariant blur kernel and obtain latent image by tradition pipeline. Chakrabarti [13] trained a neural network to predict complex Fourier coefficients of motion kernel. Recently kernel-free end-to-end deblurring methods are proposed by Nah et al. [8] and Kupyn et al. [9]. Nah [8] adopted the multi-scale network to mimic conventional coarse-to-fine optimization methods, and proposed a new realistic blurry image dataset with ground truth sharp images. The work of Kupyn [9] trains the popular Generative Adversarial Network (GAN) on the same dataset with fewer parameters, gains higher PSNR values than that of Nah et al. [8] on the GOPRO dataset, and beats the others on Kohler dataset [10] using SSIM. Although [9] performs well based on metric scores, visually, its deblurred result suffers grid artifacts, as illustrated in Fig. 1.

To address this artifact, we utilize the dark channel prior. Dark channel is defined as minimal intensity among three color channels of pixels in a local area. It was first proposed by He et al. [11] for dehazing problem, based on the statistics that haze-free outdoor images have a smaller dark channel than hazy images. Pan et al. [4] applied dark channel prior to image deblurring. They theoretically and empirically proved that comparing with blur images, the dark channel of sharp image is more sparse. And their results demonstrate that dark channel prior contributes to suppressing ringing and other artifacts. In order to enforce the sparsity, they utilize a regulation term of $L_{0}$ norm to count the nonzero elements of dark channel maps. Unfortunately, $L_{0}$ norm is not differentiable, which makes it hard to utilize in back propagation of neuron networks. Instead of using $L_{0}$ norm, we adopt $L_{2}$ norm to directly compute difference of the dark channel maps between groundtruth sharp images and deblurred images.

In this paper, we present a GAN based image deblurring network using dark channel difference as loss function. The proposed technique is not just a straightforward application of GAN. This method focuses on how to combine traditional knowledge with deep learning to make the network achieve better performance. Compared to the previous GAN-based deblurring network, the proposed network has less layers and weights. It leads to less training and testing time, and more importantly achieves favorable results. In addition, the original GOPRO training dataset consists of artificially created blurry images without noise, which are usually different from the real blurry images. To improve the quality of our trained network on more realistic blurry images and increase network robustness, we add random Gaussian noise with variance in a limited range onto the training image patches. The comparison experiments show that our network outperforms Kupyn et al. [9] for both GOPRO test dataset and real noisy blurry images.

Related Work

Conditional General Adversarial Networks

GAN is first proposed by Goodfellow et al. [14] to train a generative network in an adversarial process. It consists of two networks: a generator $G$ and a discriminator $D$. Generator generates a fake sample from input noise $z$, while discriminator estimates the probability that the fake sample is from training data rather than generated by generator. These two networks are simultaneously trained until discriminator cannot tell if the sample is real or fake. This process can be summarized as a two-player min-max game with the following function:

[TABLE]

where $\text{P}_{\text{data}}$ denotes distribution over training data $\bar{x}$ and $\text{P}_{z}$ is distribution of input noise $z$. GAN has been applied to different image restoration problems like super-resolution [16] and texture transfer [17].

Mirza et al. [15] extend GAN into a conditional model (eq. (2)), called Conditional Generative Adversarial Nets (CGAN), so that GAN can make use of auxiliary information to direct both generator and discriminator. Isola et at. [18] adopt CGAN architecture to achieve general image-to-image translation. In [18], more than just random noise $z$, similar image $y$ is added as input of the generator, where $y$ and $\bar{x}$ share part of features. $y$ and $\bar{x}$ can be pairs of hazing and clear images about same scene, or different color buildings with same structure. Based on network architecture of [18], Kupyn et al. [9] utilize Wasserstein loss [19] and perceptual loss [20] to train a CGAN for deblurring problem.

[TABLE]

Dark Channel Prior

For an image $I$, the dark channel of a pixel $p$ is defined by He et al. [11] as

[TABLE]

where $p$ and $q$ are pixel locations, $\mathcal{N}(p)$ denotes the image patch centered at $p$, and $I^{c}$ is the $c$-th color channel. As shown in eq. (3), dark channel describes the minimum intensity in an image patch. He et al. [11] observe that dark channel map $D(I)$ in a haze-free image tends to be zero. Pan et al. [4] use a less restrictive assumption that dark channel map $D(I)$ is sparse rather than zero. Inspired by this, they adopt $L_{0}$-regulation term to enforce less sparse dark channel in a deblurring process, where $L_{0}$ norm counts non-zero elements in a dark channel map.

Proposed Method

Network Architecture

The proposed network aims at obtaining a generator to restore sharp image $I_{S}$ from input blurry image $I_{B}$. This generator is trained with a discriminator using pairs of blurry image $I_{B}$ and ground truth sharp image $I_{\bar{S}}$. This structure is shown in fig.2. Except for the first layer of discriminator and generator, each block in both generator and discriminator consists of a convolutional layer, batch normalization step [21] and an activation function LeakyReLu [23] with leaking rate $\alpha=0.2$. The first layers are not normalized.

Generator The proposed generator adopts an encoder-decoder framework to achieve image-to-image performance. Similar to [18], the encoder consists of a sequence of convolutional layers with stride $=2$ and kernel size $=5$. And the decoder has a chain of transposed-convolutional layers with same size of stride and kernel. Encoder represents the input image with a bottleneck vector and decoder recovers an image with same size of input from bottleneck vector. A skip architecture is applied by inserting same size of layers from encoder after each layer of decoder. This skip connection refines the details in output image by combining deep, coarse, semantic information and shallow, fine, appearance information [22]. Dropout is also included in decoder to avoid over-fitting.

Discriminator The proposed discriminator contains a series of convolutional layers with stride $=2$ and kernel size $=5$. The output of discriminator is a scalar, followed by a sigmoid function.

Loss Functions

According to eq. (2), we train discriminator and generator alternatively. The loss function of discriminator is same as adversarial loss:

[TABLE]

In the deblurring setting, $y$ and $\bar{x}$ denote blurry and sharp image, respectively. The generator loss is defined as combination of adversarial loss, content loss and dark channel loss:

[TABLE]

where $\lambda_{1}=100$ and $\lambda_{2}=250$ in our experiments.

Content loss We adopt the traditional content loss to direct the output of generator to ground truth. Although both $L_{1}$ and $L_{2}$ norm are commonly used, $L_{1}$ norm is chosen since it attains less blurring result [18].

[TABLE]

Dark channel loss In order to suppress ringing and grid artifact, dark channel prior is especially chosen. Pan et al. [4] exploit $L_{0}$ norm to count non-zero elements in a dark channel map $\mathfrak{D}_{c}(I)$ of an image $I$. Since $L_{0}$ norm is indifferential, $L_{2}$ norm is utilized instead which calculates the distance of dark channel map between ground truth and deblurred image.

[TABLE]

Unlike [9], we discard the perceptual loss [20]. Kupyn et al. [9] employ the difference of one feature map in the VGG-19 [24] between ground truth and restored images as perceptual loss. GAN is known for its ability to reserve perceptual feature of an image. Adding an extra perceptual loss seems a noneffective repeat. Our experiment shows that perceptual loss doesn’t improve the result, on the contrary, it leads to worse performance.

Experiments

Our network is implemented with Python code based on Tensorflow [25].

Datasets

GOPRO dataset [8] is utilized for training and testing our network. It contains 2103 paris of blurry and ground truth images in train dataset, and 1111 pairs in test dataset. Resolution of the image are 720p. The blurry image is generated by averaging a sequence (7-15) of continuous sharp images. Sharp image in the middle of sequence is regarded as ground truth. GOPRO dataset is regarded as benchmark by many deblurring algorithms like [8] and [9]. Although GOPRO dataset is widely used, it only employs noise-free images. For natural images, however, noise always accompanies with blur. To test our model on more real images, we add Gaussian noise with $variance=0.001$ to original GOPRO_Large dataset and create a new GOPRO-noise dataset with 1111 image pairs. A synthetic dataset in [9] is adopted for training. Same as combination version of DeblurGAN in [9], we use both GOPRO train dataset and synthetic dataset to train our network.

Training Process

The proposed network is trained on NVIDIA GeForce GTX 1080 Ti GPU and tested on Mac Pro with 2.7 GHz Intel Core i5 CPU. Similar to [9], the input training pair is randomly croped as size of $256\times 256$ after downsampled by a factor of two. Weights are initialized to follow Gaussian distribution with zeros mean and standard deviation $0.02$. For each iteration of optimization, 1 step is performed on discriminator $D$, followed by 2 steps on generator $G$ to prevent discriminator loss $\mathscr{L}_{d}$ from zero. The model is trained for 15 epochs within 2 days, comparing with 200 epochs for 6 days in [9] . Furthermore, despite of instability GAN’s training, our method converges to similar result for each and every training task, which demonstrates the robustness of our GAN architecture.

Result and Comparison

Our test results are mainly compared with state-of-art GAN based deblurring network DeblurGAN [9]. DeblurGAN defeats deep learning networks [7] and [8] on GOPRO dataset. Since the author posted the code online111https://github.com/KupynOrest/DeblurGAN, we compare our network with DeblurGAN by directly adopting its uploaded network and latest trained weights. We test our model on GOPRO and GOPRO-noise test datasets.

Fig. 3 illustrates the deblurred results of [9] and our model. Blurry image in the first row is picked in GOPRO-noise dataset and the blurry one of second row is real natural image with motion blur taken by camera. According to local patches, although [9] can deal with blur but its results suffer from grid artifacts, while our model with dark channel loss achieves sharper images without grid artifacts. Furthermore, for motion blurry image (second row), the sharp part in input image remains unchanged in our deblurred result, but extra grid artifacts are added to result of [9].

The quantitative performance of the proposed network on two dataset GOPRO and GOPRO-noise is shown in Tab. 1. In our experiment, the coefficient of dark channel loss $\lambda_{2}=250$($dc250$). The results are compared with same network without dark channel loss $dc0$, same network with extra perpetual loss $dc250p$ as well as DeblurGAN [9]. All test images are downsampled by factor of two. The perpetual loss follows what it is in [10]. The proposed model performs best among the comparisons for both noise-free and noisy dataset. DeblurGAN performs less well owing to its grid artifacts. Perceptual loss leads to a worse result. Since GAN is good at preserving perceptual feature already, perceptual loss brings no extra constraints for the network. Comparison with dc=0 demonstrates that dark channel loss contributes to better result.

Conclusion

To address deblurring problem using a CGAN based architecture and to tackle the issue with grid artifacts in GAN based deblurring methods, this paper incorporates a dark channel prior. The dark channel prior is employed by $L_{2}$ norm rather than $L_{0}$ in order to make it more friendly for network training. To validate the deblurring result on more nature images, a noise involved dataset is proposed. The proposed network shows a great deblurring performance for both synthetic and real blurry images.