Integrating neural networks into the blind deblurring framework to compete with the end-to-end learning-based methods

TL;DR

This paper proposes a hybrid blind deblurring approach that combines neural networks with traditional methods, improving detail restoration and generalization over pure deep learning models.

Contribution

It introduces a novel integration of CNNs into the deblurring framework, specifically with SEN and RP-GAN, to address limitations of end-to-end neural methods.

Findings

Outperforms state-of-the-art end-to-end methods in detail restoration.

Shows better generalization to complex motion blur.

Restores more reasonable image details.

Abstract

Recently, end-to-end learning-based methods based on deep neural network (DNN) have been proven effective for blind deblurring. Without human-made assumptions and numerical algorithms, they are able to restore images with fewer artifacts and better perceptual quality. However, in practice, we also find some of their drawbacks. Without the theoretical guidance, these methods can not perform well when the motion is complex and sometimes generate unreasonable results. In this paper, for overcoming these drawbacks, we integrate deep convolution neural networks into conventional deblurring framework. Specifically, we build Stacked Estimation Residual Net (SEN) to estimate the motion flow map and Recurrent Prior Generative and Adversarial Net (RP-GAN) to learn the implicit image prior in the optimization model. Comparing with state-of-the-art end-to-end learning-based methods, our method restores reasonable details and shows better generalization ability.

Figures

Tables

Table 1. Table 1: MSE on motion flow estimation

Dataset	Sun et al.	Gong et al.	Ours
MSCOCO-23	24.34	5.54	2.71
MSCOCO-46	63.37	7.21	4.24

Table 2. Table 2: Quantitative results on GOPRO testing dataset

Method	Sun et al.	Nah et al.	Tao et al.
PSNR	24.64	29.08	30.10
SSIM	0.8429	0.9135	0.9323
Time	20min	3.09s	1.6s
Method	Ours.it1	Ours.it2	Ours.it3
PSNR	30.03	30.96	31.25
SSIM	0.9352	0.9487	0.9518
Time	17s	32s	47s

Table 3. Table 3: Quantitative results on Kohler dataset

Method	Sun et al.	Nah et al.	Tao et al.
PSNR	25.22	26.48	26.80
SSIM	0.7735	0.8079	0.8375
Method	Ours.it1	Ours.it2	Ours.it3
PSNR	27.25	28. 46	29.19
SSIM	0.8642	0. 8933	0.9128

Table 4. Table 4: Comparison of different recurrent levels

Level	1	2	3	4	5
PSNR	28.77	29.58	29.86	29.83	29.84
SSIM	0.9273	0.9305	0.9314	0.9312	0.9312

Equations

$${\mathcal{L}}_{L_2}=\frac{1}{N}\Vert U^{\ast }-U^l{\Vert }_2^2$$

$$I=\arg mi{n}_I\Vert I\ast K-O{\Vert }_2^2+\gamma p(I)$$

$$\left\{\begin{array}[]{ll}\begin{split}\ I^{*}_{n+1}=&\arg min_{I^{*}_{n+1}}\frac{\beta}{2}\|I^{*}_{n+1}-Z_{n}\|^{2}_{2}\\ &+\frac{1}{2}\|I^{*}_{n+1}*K-O\|^{2}_{2}&{\raisebox{0.5pt}{\raisebox{-.9pt} {1}⃝}}\\ \ Z_{n+1}=&\arg min_{Z_{n+1}}\frac{\beta}{2}\|I^{*}_{n+1}-Z_{n+1}\|^{2}_{2}\\ &+\gamma p(Z_{n+1})&{\raisebox{0.5pt}{\raisebox{-.9pt} {2}⃝}}\end{split}\\ \end{array}\right.$$

$$I_{n+1}^{\ast }=[K^{T}K+\beta \mathcal{I}{]}^{-1}[\beta Z_n+K^{T}O]$$

$$\begin{array}{rl}\arg \underset{G}{\min }\underset{D_{cGAN}}{\max }& {\mathbb{E}}_{I_s,O}[log(D_{cGAN}(I_s|O))]+\\ & {\mathbb{E}}_{I^{\ast },O}[log(1-D_{cGAN}(G(I^{\ast }|O)|O)]\end{array}$$

$$W(P_{data},P_o)\approx \underset{||D_p||\le 1}{\max }{\mathbb{E}}_{x\sim P_{data}}[D_p(x)]-{\mathbb{E}}_{y\sim P_o}[D_p(y)]$$

$${\mathcal{L}}_p(G,D_p)=(D_p(G(I^{\ast }|O))-D_p(I_s){)}_{+}$$

$$\begin{array}{rl}{\mathcal{L}}_{RP-GAN}=& {\mathcal{L}}_c(G)+\gamma {\mathcal{L}}_{cGAN}(G,D_{cGAN})+\\ & \lambda {\mathcal{L}}_P(G,D_p)\end{array}$$

$${\mathcal{L}}_{cGAN}(G,D_{cGAN})=log(1-D_{cGAN}(G(I^{\ast }|O)|O))$$

$${\mathcal{L}}_c(G)=\frac{1}{C\ast H\ast W}\sum _{c=1,i=1,j=1}^{C,H,W}[\varphi (G(I^{\ast },O))-\varphi (I_s)]$$

Full text

1 Introduction

Motion blur is a commonly appeared degradation of image qualities. The blurry images generally caused by the shakes of camera and fast object motions. The problem is highly ill-posed due to the unknown blur kernel and extra noise.

The end-to-end learning based methods solve it by training a neural network to restore the clear images from blurred observations directly. Depending on the strong fitting ability of deep neural network (DNN), they can restore images with fewer artifacts and better visual effect. Previous work [18, 16, 27, 10] had attempted various kinds of neural networks, including Residual Network (ResNet), Multi-scale Convolutional Neural Network and Generative and Adversarial Net (GAN). These work proved deep and well-designed network structure is the key to success. However, in practice, we also find some limitations of this kind of methods. Firstly, most of them can’t perform well when the motion is too large or the blur is highly space-variant. Besides, sometimes they restore heavy blurry content to unreasonable objects since lacking theoretical guidance. The relevant results are shown in our experiment.

On the other side, conventional deblurring methods generally focus on getting the better blur kernel and image prior. The development of the neural network encourages many attempts to embed the neural network into the conventional deblurring framework. Utilizing the neural networks to estimate the blur kernel seems to be straightforward and effective [25, 6, 21, 32], while learning image prior is somewhat challenging. One way to do so is learning the implicit image prior directly from ground truth [34, 4, 22, 3]. A typical strategy of this kind of methods is using half-quadratic splitting algorithm to split the non-blind restoration task to two sub-problems and then alternatively solve them to get the final solution. The deconvolution sub-problem can be solved by numerical method directly. Another denoise sub-problem, which contains the unknown image prior, is solved by the neural network. However, it is not easy to build such a neural network which can adapt to the iterative optimization process well. Zhang et al. [34] pre-trained multiple CNN for image denoising task and integrated them into the optimization-based framework. However, these pre-trained parameters can not dynamically adjust with the iterative optimization. Dong et al. [4] computed the inexact solution of the deconvolution sub-problem with a single step of gradient descent. Although it enables the whole optimization model to be trainable from end to end, it also takes extra optimization steps. Moreover, since both of them [34, 4] are trained for non-blind restoration, they are hard to be applied to blind deblurring directly.

We can see either the end-to-end learning methods or the optimization-based methods have their drawbacks. However, end-to-end learning deblurring methods are more often applied in practice recently considering their higher comprehensive competence—their better visual effects, higher speed and complete blind deblurring models which make them can be conveniently used. But in this paper, we find optimization-based methods have the potential to outperform the end-to-end learning based methods and make up for their shortcomings. We design our optimization part following the implicit-prior-learning strategy, but we build a Generative and Adversarial Net(GAN) with deeper layers and higher ability to generate more realistic images. We introduce the recurrent structure in the network and enable it can be trained online with the iterative optimization process. We call the network as Recurrent Prior Generative and Adversarial Net (RP-GAN). For blind deblurring, we build a stacked residual network, called Stacked Estimate Net (SEN), to estimate the space-varying motion flow of the blurred image. Although the estimation strategy is straightforward, its results are the best comparing with other similar methods. Furthermore, in order to ensure the robustness of RP-GAN to the inexact blur kernels, we add an extra discriminator to impose extra punishment on artifacts caused by the incorrectly-estimated kernels. In the end, we send the result to the whole system again to further process the residue blur in global iteration. Our intact blind deblurring model is shown in Figure 2. To our knowledge, this is the first optimization-based deblurring method comparing with the powerful end-to-end learning based methods in the experiment and getting the better performance.

In conclusion, the main contributions of this paper are:

• •

We build SEN to estimate the nonuniform motion flow map. Its accuracy exceed all the other deep learning based methods.

• •

We propose RP-GAN and integrate it into the iterative optimization-based framework well.

• •

We design the aforementioned two parts to be well-compatible, thus making the integrated blind deblurring model to be robust and achieve better deblurring effect.

2 Related Work

Most conventional optimization-based deblurring approaches get success depending on guiding maximum a posteriori probability (MAP) process by assumed priors, such as the total variational regularizer [29, 20], Gaussian scale mixture priors [5], normalized sparsity [9], L0 gradients [30], dark channel prior [19], etc. However, since all these hand-crafted priors are designed under limited observations or restricted assumptions, these algorithms show unsatisfied results when processing the images that have fewer corresponding features.

The development of neural networks inspires many works to utilize it in conventional deblurring framework. Some work focus on learning the blur kernel [21, 32, 25, 6]. Sun et al. [25] proposed a convolutional neural network (CNN) to predict the probabilistic distribution of blur. Gong et al. [6] proposed a fully-convolutional deep neural network (FCN) to directly estimate the blur kernel in pixel level and get a higher accuracy. On the other side, several work learn generic image priors (explicit or implicit) for optimization [23, 34, 22, 33, 12, 2, 4, 3]. In terms of learning explicit prior, Zhang et al. [33] learned denoised gradient as image prior to guide the deconvolution model. Li et al. [12] learned a discriminative prior for deblurring in generic scenarios. However, they have to compromise on restoring quality when embedding an explicit prior to numerical methods. As for learning implicit prior, Zhang et al. [34] used pre-trained multiple CNN to restore the corrupted deconvolutional results. Dong et al. [4] made a compromise on deconvolutional solutions to make the whole optimization model can be trained from end to end. Nevertheless, these operations are lossy. Besides, these methods are hard to be applied to blind deblurring directly. Furthermore, their networks are actually too simple to fit their tasks.

Recently, end-to-end learning based methods have shown great advantages for deblurring. Nah et al. [16] proposed a multi-scale CNN for restoring the images in three different levels. Each level processes different blur scale, from small to large. Kupyn et al. [10] built GAN model for deblurring directly from the blurred observations to the sharp images. Tao et al. [27] inherited the multi-scale structure from Nah et al. [16] and added long short-term memory (LSTM) in the recurrent process. End-to-end learning based methods restore the images with fewer artifacts than optimization-based methods. But they are highly dependent on the observed data. Thus, on the one hand, they often show poor generalization ability, especially when the motion is large or complex. On the other hand, since these methods completely discard the deblurring principle, they sometimes restore unreasonable contents from blurred images, which may cause the restored images to be more confusing than the blurred ones.

3 Motion Flow Estimation by SEN

We propose a stacked residual network to estimate spatially-varying motion flow map from end to end. Our motion flow map is modeled following [6]. Specifically, given an arbitrary RGB blurry image $O$ which has the size $H*W$, the task is generating a $H*W*2$ matrix $U$ to represent motion flow map. $U$ can be expressed as : $U(i,j,1)=u_{i,j}$ , $U(i,j,2)=v_{i,j}$, $\forall i,j\in O$. $u$ and $v$ denote horizontal and vertical motions respectively.

The network we propose, called SEN, is mainly based on an encoder-decoder structure which has been proven useful in various vision tasks [14, 24, 26, 31, 17]. We adapt the network structure from [17]. The detailed structure is shown in Appendix A. Unlike the symmetrical structure used in [17], we use three dilated convolution layers with increasing channels in front of the network. They help to enlarge the receptive field and abstract deep blur features .

We also use stacked structure to improve the estimation accuracy. Stacked structure takes the result of the last network as the input of the next one. In the paper, the stacked networks share the similar structures. The result from the first network will be concatenated with the blurred image to serve as the next network’s input. Although more stacks can further improve the estimation accuracy, they will also cause over-fitting. For the balance, we take two stacks in practice.

In previous work [25, 6], the task was treated as a classification problem. However, we find using regression loss can reach higher accuracy. Specifically, L2 loss is used for training, which can be expressed as :

[TABLE]

where $U^{*}$ denotes the estimated matrix and $U^{l}$ denotes the label. $N$ is the number of elements in the matrix.

4 RP-GAN embedded in HQ Algorithm

4.1 Half-Quadratic Splitting Algorithm(HQ Algorithm)

Once knowing the motion flow map, the deblurring issue can be modeled as:

[TABLE]

where $I$ denotes the latent sharp image. $O$ denotes the given blurry image. $K$ denotes the heterogeneous motion blur kernel map with different blur kernels for each pixel in $O$. $p(I)$ denotes the latent prior of the image. $*$ implies general convolution operation. $\gamma$ is a weighting constant. Each nonuniform blur kernel in map $K$ will be applied to its corresponding pixel. The specific operation is following [6]. In Eqn. (2), since both $I$ and $p$ are unknown, it’s hard to solve the equation directly.

Half-quadratic splitting algorithm is a way to simplify it. By HQS, we can split Eqn. (2) into two sub-problems which show in Eqn. (3).

[TABLE]

where $n$ is the number of iterations. $I^{*}$ is the corrupted deconvolutional image, $Z$ is an auxiliary variable initialized with observed blurry image $O$. $\beta$ is a variable parameter. Eqn. (2) can be solved by alternatively solving two sub-problems with increasing $\beta$. In Eqn. (3), Eqn. (3)-\raisebox{-.9pt} {1}⃝ has the analytic solution :

[TABLE]

where $\mathcal{I}$ is the identity matrix. But generally, the inverse matrix in it is hard to be computed directly. In practice, we use conjugate gradient algorithm to solve it.

Eqn. (3)-\raisebox{-.9pt} {2}⃝ is a denoising module containing the unknown image prior, thus can not be solved without any assumption. Therefore, training a neural network to solve Eqn. (3)-\raisebox{-.9pt} {2}⃝ is an appealing idea. In this way, we can learn the implicit image prior directly through data.

4.2 Recurrent Prior GAN and Insights behind it

For using neural networks to solve Eqn. (3)-\raisebox{-.9pt} {2}⃝ in different iterative steps, a straightforward way is training multiple neural networks to solve each of them separately [34]. However, [4] has explored the possibility that using one deep neural network to fit all these different modules. This point also matches our observation: a recurrent network will output clearer images when accepting better-quality deconvolutional results, which makes us believe network could learn to adjust the latent parameter $\beta$ by itself (through perceiving variant inputs). So we build Recurrent Prior GAN (RP-GAN) to learn Eqn. (3)-\raisebox{-.9pt} {2}⃝ from end to end. The basic structure of RP-GAN is roughly based on [7]. We provide the details of it in Appendix A. In recurrent process, the restored result of the $n$ recurrence (level) is served as variable $Z_{n}$ for Eqn. (3)-\raisebox{-.9pt} {1}⃝, then it will be used to calculate the corrupted deconvolutional value $I^{*}_{n+1}$ through Eqn. (3)-\raisebox{-.9pt} {1}⃝. $I^{*}_{n+1}$ will then be the input of $n+1$ level in the recurrent generator. The process is shown in Figure 3. In the implementation, such a process will run three times. In another word, RP-GAN contains three levels.

However, although building one network with recurrent structure can prevent over-fitting, it also raises another problem: the training process will suffer heavy oscillation since the variance of RP-GAN will cause the change of its own inputs in the next level. For preventing oscillation, we do not optimize RP-GAN immediately once a single deconvolutional result is obtained. We set a buffer to store deconvolutional results of different levels produced by a frozen generator. Considering the results of the front levels are more influential than the latter ones, we make the results of the front levels occupy a larger proportion of the buffer. After getting a certain number of samples, we then renew and reinforce RP-GAN on the buffer by randomly feeding these samples. This strategy is somewhat like the widely-used experience replay method and fixed Q-targets in reinforcement learning.

Another trouble occurred in our training is that, if we use the blur kernels from SEN, the results will show many artifacts even after sufficient training (we show these results in Appendix B). This is because the estimated blur kernels are not completely correct. Even tiny errors of the kernels will cause amplified artifacts. This problem could not be autonomously addressed through the network’s supervised learning because once network outputs the images with remained artifacts, the artifacts will then escalate in the next deconvolution process and lead the inputs of the next level become even worse. The artifacts’ corruption will then be gradually aggravated in the process of iteration. The way we solve that is forcing the intermediate results to be optimized from blurry to clear but not from artifacts-corrupted to clear. Specifically, we add an extra artifacts regularization term with the form of discriminator in RP-GAN, called artifacts-penalized discriminator. The discriminator regards blurred image and artifacts-corrupted image as two extremes. It imposes extra punishment to the generator when the generated image is closer to the artifacts end while imposes no punishment when it is closer to the blur end. The blurry content is generated by feeding the original blurred image to the generator for compensating the artifacts. Since the deconvolution model will trust the blurry intermediate result more in the next iteration (as parameter $\beta$ has increased), it will produce fewer artifacts than the previous input, so that the image will become clearer and clearer after iterations. We show the variety of intermediate results in Figure 4. Next we will look at the generator and the discriminators in detail.

4.2.1 Generator

The inputs of the generator is the deconvolutional image: $I^{*}$ and the concatenated observed blurred image: $O$, which is used to compensate the heavily corrupted information in $I^{*}$. We also skip connect the concatenated inputs with the last layer of the network to make the network to be more dependent on the offered information, thus to be sensitive to different inputs in different iterations.

4.2.2 Discriminator

Two discriminators are built to ensure RP-GAN to recover images from blurry to clear. cGAN discriminator plays a minimax game with $G$, to penalize the difference between generated images and sharp images. Artifacts-penalized discriminator gives extra penalty to the artifacts. Two discriminators help $G$ restore the clear content with the fewest artifacts.

cGAN discriminator Since we have used the blurred image as additional information, we build our generative and adversarial strategy following conditional generative and adversarial net [15]. The objective of $G$ and $D$ in it can be expressed as:

[TABLE]

where $I_{s}$ denotes the latent sharp image. $O$ denotes the observed blurry image. $I^{*}$ denotes the corrupted deconvolutional image. Generator $G$ learns to minimize the objective against discriminator ${D_{cGAN}}$, which tries to maximize it. The generator is expected to generate images closer to natural ones through playing the minimax game. The generated image will be sent to the discriminator with concatenated conditional variable. By comparing the blurred image with the generated image, cGAN discriminator can better distinguish the generated one from the sharp one.

Artifacts-penalized discriminator For constraining artifacts, we build an extra penalty term. Artifacts-penalized discriminator penalizes $G$ if generated image is closer to the corrupted input than the sharp image. We adapt WGAN [1] to our discriminator. WGAN discriminator is trained to approximate the Wasserstein distance, or called Earth-Mover distance, between the generated distribution and the real distribution. But here, we train the artifacts-penalized discriminator as a Wasserstein distance between the distribution of corrupted input and the distribution of observed blurred image. The objective of artifacts-penalized discriminator can be expressed as :

[TABLE]

where $P_{data}$ is the distribution of the corrupted input data and $P_{o}$ is the distribution of the observed blurred image. $x$ and $y$ are the samples from distribution $P_{data}$ and distribution $P_{o}$ respectively. $W(P_{data},P_{o})$ denotes the Wasserstein between two distributions. $D_{p}$ denotes the artifacts-penalized discriminator. $||D_{p}||\leq 1$ denotes that $D_{p}$ should be 1-Lipschitz function. The penalty to $G$ is expressed as:

[TABLE]

$()_{+}$ denotes the positive part of the content. As shown in Eqn. (6), artifacts-penalized discriminator is trained to output a larger value for artifacts-corrupted image and a smaller value for blurry image. Thus, as shown in Eqn. (7), punishment is only imposed when $D_{p}(G(I^{*}|O))$ is larger than $D_{p}(I_{s})$, which denotes the generated image is closer to the corrupted input than the sharp image in learned measure.

4.2.3 Loss function

Total loss of RP-GAN is expressed as:

[TABLE]

where $\mathcal{L}_{c}(G)$ denotes the content loss. $\mathcal{L}_{cGAN}$ denotes the adversarial loss. $\mathcal{L}_{P}$ denotes extra penalty. $\lambda$ and $\gamma$ are weighting constants. The objective of RP-GAN is simultaneously minimizing the three items until convergence.

Specifically, adversarial loss can be expressed as:

[TABLE]

Our content loss is perceptual loss [8], which can be expressed as:

[TABLE]

where $\phi$ is a pre-trained network for abstracting the depth value of images. $C,H,W$ are the channel, height and weight of a hidden layer of network $\phi$. Perceptual loss compares the depth value of images instead of directly penalizing the difference of pixel-level information. In our experiment, it helps to prevent the network from over-fitting to the image contents and generate images with better perceptual quality. In practice, we compute the loss at layer relu2_2 of the VGG16 network. Extra penalty $\mathcal{L}_{P}$ is shown in Eqn. (7).

5 Global Iteration

In experiment, we find sending the final result to the whole model again can further recover the residual blur. This effect is more apparent on more challenging datasets (GOPRO dataset [16] and Kohler dataset [11]) than the simpler one (images generated by linear blur kernels). We speculate that this may be because after the blurry image first passes through our model, large and complex blur is segmented into small and simple residual blur which is easy to be accurately estimated and further handled. Although we can’t fully explain it yet, we use it to improve our results. The quantitative comparison of the effect is shown in our experiment section.

6 Experiments

Our networks are implemented using PyTorch deep learning framework. All models are ran on NVIDIA 1080Ti GPU.

6.1 Motion Blur Kernel Estimation

Datasets Our data is generated following [6]. For adapting to different scales of blur kernels, we generate two datasets. One is used to estimate small-scale blur kernels, the ceiling is set as $v_{max}=u_{max}=23$. The other is used to estimate large-scale ones, the ceiling is set as $v_{max}=u_{max}=46$. Both datasets contain 12000 blur images generated from 1200 sharp images from Microsoft COCO [13] (Each sharp image generates 20 blur images). The 10000 images of the datasets are used for training. The other 2000 are used for testing. Two datasets are referred as MSCOCO-46 and MSCOCO-23 respectively.

Training details SEN is trained by images cropped to 256*256 pixels. We set two stacks in practice. Both two stacks are trained by batch size 16. In the first stack, learning rate is set as $1.25\,e^{-3}$ in the first 50 epochs, and linearly decreases to $1.25\,e^{-5}$ in the next 40 epochs. The second stack is trained 15 epochs by learning rate $1.25\,e^{-5}$. Both of them use Adam solver with $\beta_{1}=0.9$, $\beta_{2}=0.99$ and $\epsilon=10^{-8}$.

Comparison We compare our method with Sun et al. [25] and Gong et al. [6]. Sun et al. estimated blur kernel in patch by CNN first, then further processed them with markov random field for smoothness. Gong et al. proposed a FCN to estimate motion flow maps in pixel level and got previous state-of-the-art results. In our experiments, SEN shows higher qualitative results than previous networks by comparing mean square error (MSE). The comparison is shown in Table 1.

6.2 Blind Deblurring

For evaluating our blind deblurring method, we compare our method on mainstream benchmarks with previous state-of-the-art image deblurring approaches. PSNR and SSIM metrics are used to evaluate the restored image quality.

RP-GAN training details RP-GAN is trained on two different datasets to ensure model’s generalization. One contains 1000 synthetic blurred images generated following section 6.1. The other is GOPRO dataset [16], which provides 2103 blur/clear pairs for training. All deconvolutional results are solved by the estimated blur kernels from the pre-trained SEN model. Batch size is set as 16. Learning rate is set as $1e^{-4}$ for both generator and discriminator in the first 110 epochs, and linearly decreases to $1e^{-6}$ in the next 80 epochs. We use the same solver as which used for SEN. Before the recurrence begin, we pre-train RP-GAN on the first level for 40 epochs. Then we set a buffer which stores 1000 images including 50% first-level results, 30% second-level results and 20% third-level results. These images are randomly sent to RP-GAN for training 3 epochs before the buffer updated. The whole training process costs about 15 days.

Comparisons on benchmarks We compare our algorithm with [25] [16] and [28]. Sun et al. [25] learned blur kernel through CNN, then did non-blind deblurring by traditional deconvolution method. Nah et al. [16] and Tao et al. [28] are two representative end-to-end learning methods for deblurring. Both of them used multi-scale strategy and generated perceptually convincing images. Tao* et al*. got previous state-of-the-art results on mainstream benchmarks.

Methods are compared on two mainstream benchmarks, GOPRO dataset [16] and Kohler dataset [11]. GOPRO dataset generated long-exposure blurry frames by averaging consecutive short-exposure frames from videos captured by high-speed cameras. The dataset provides 2103 clear/blur pairs for training and 1111 pairs for testing. The quantitative results on GOPRO testing dataset are listed in Table 2. Visual comparison is shown in Figure 5. More comparisons are provided in Appendix B.

In Table 2 and Table 3, it 1,2,3 denote the results of global iteration 1, 2 and 3 respectively. We see the results are improved with the iteration going on, and get the best result in the third iteration. Because containing the deconvolution process, our method will certainly cost more time than end-to-end learning methods. But using the deep neural network for optimization helps us take much less time than Sun et al., who used a numerical method.

Kohler dataset [11] recorded and analyzed real camera motion, which is played back on a robot platform. The dataset consists of 4 images blurred with 12 different kernels for each of them. Comparisons on Kohler dataset are listed in Table 3. Visual comparison is shown in Figure 6.

Comparing with end-to-end learning methods, our method restores clearer details on GOPRO test dataset. On Kohler dataset, our method outperforms others by a large margin. That is because the restoration ability of end-to-end learning based methods is only dependent on training dataset. It results in their failure of restoring extreme motions. This fault seems not so apparent on GOPRO testing dataset. But sometimes it restores unreasonable objects when the blur content is heavy. Such as Nah et al. restore the slant license plate number in Figure 5. Unlike the end-to-end learning methods, we build two networks independently estimate the motion flow and restore the deconvolutional results in the optimization framework. Because of learning a easier subtask, SEN can have a better awareness of motion than the end-to-end deblurring networks. Then RP-GAN can utilize the known motions to restore more reasonable objects. A typical example is shown in Figure 7. On the other hand, end-to-end learning based methods perform worse on challenging blur images in Kohler dataset, which the motion is larger or more complex than which in GOPRO dataset. That’s because their networks haven’t ’seen’ this pattern of motion. But since the deblurring framework we use is universally applicable, our method shows stronger generalization capability.

** Recurrent strategy** The number of the recurrent levels may also be a factor influencing the result. We compared the effect of setting different levels. The quantitative results of each setting are listed in Table 4. We see the results improve a lot in the first three levels, and converges later. For the best efficiency, we take three levels in the paper.

The effectiveness of artifacts-penalized discriminator RP-GAN was trained without artifacts-penalized discriminator at first. The inexact blur kernels cause the results show many artifacts. Visual comparison is shown in Appendix B.

Comparison on real blurred images The comparisons with other methods on real-captured images are provided in Appendix B.

7 Conclusion

In this paper, we design the better neural networks to estimate the blur kernels and learn the implicit image prior, then further integrate the two parts to build an effective and convenient blind deblurring model. The experimental results show the effectiveness of the proposed method, which also proves that an optimization method based on theoretical guidance can overcome some shortcomings of the end-to-end learning based methods and get better results. We believe there still have many possibilities worth exploring about this method. For example, estimating another form of the blur kernel, more experiments and analyses of global iteration, simplifying the deconvolution operation then jointly optimizing the estimation part and the optimization part, etc. We are excited about these possibilities and would like to further investigate them in our future work.

Appendix A RP-GAN structure & SEN structure

We use CBR $k*k*c:s$ denote the convolution layer with batch normalization and relu activation function, the convolution layer has kernel size $k*k$ and $c$ output channels applied with stride $s$ , use** Res1** $c$ denote the first kind of Residual Block which has $c$ output channels, Res2 $c$ denote the second kind of Residual Block which has $c$ output channels, then has:

Res1 $c$ : $input\to CBR\;3*3*c:1\to CBR\;3*3*c:1+input\to output$

Res2  $c$ : $input\to CBR\;3*3*c/2:1\to CBR\;3*3*c/2:1\to CBR\;3*3*c:1+input\;\divideontimes\;CBR\;3*3*c:1\to output$

RP-GAN

Generator : $input\to CBR\;7*7*64:1\to CBR\;3*3*128:2\to CBR\;3*3*256:2\to Res1\;256\to Res1\;256\to Res1\;256\to Res1\;256\to Res1\;256\to Res1\;256\to Res1\;256\to CBR\;3*3*256:2\to CBR\;3*3*128:2\to CBR\;7*7*64:1+input\divideontimes Res2\;3\to output$

DcGAN : $input\to CBR\;4*4*64:2\to CBR\;4*4*128:2\to CBR\;4*4*256:2\to CBR\;4*4*512:2\to CBR\;4*4*512:1\to CBR\;4*4*1:1\to sigmoid\to output$

Dp : $input\to CBR\;4*4*64:2\to CBR\;4*4*128:2\to CBR\;4*4*256:2\to CBR\;4*4*512:2\to CBR\;4*4*512:1\to CBR\;4*4*1:1\to output$

**SEN

Stack 1** : $input\to CBR\;7*7*64:1\to CBR\;5*5*128:2\to CBR\;5*5*256:2\to maxpool\;2*2\to Res2\;256\to maxpool\;2*2\to Res2\;512\to maxpool\;2*2\to Res2\;512\to maxpool\;2*2\to Res2\;512\to upsample\;2*2\to Res2\;512\to upsample\;2*2\to Res2\;512\to upsample\;2*2\to Res2\;512\to upsample\;2*2\to Res2\;256\to upsample\;2*2\to Res2\;128\to Res2\;64\to Res2\;2\to output$

The details of SEN is shown in Figure 8.

Appendix B Visual comparisons

B.1 Comparison on GOPRO dataset

B.2 The effectiveness of artifacts-penalize discriminator

B.3 Comparison on real blurred images