Blur Removal via Blurred-Noisy Image Pair

TL;DR

This paper introduces a kernel-free deblurring approach using a blurred-noisy image pair, leveraging Gaussian mixture models and optical flow to effectively restore sharp images in complex blur scenarios.

Contribution

A novel deblurring method that does not require blur kernel estimation, utilizing image pairs with different noise and shutter speed characteristics.

Findings

Outperforms state-of-the-art methods in robustness and visual quality

Effective in handling complex space-variant and space-invariant blur

Validated on synthetic and real-world datasets

Abstract

Complex blur such as the mixup of space-variant and space-invariant blur, which is hard to model mathematically, widely exists in real images. In this paper, we propose a novel image deblurring method that does not need to estimate blur kernels. We utilize a pair of images that can be easily acquired in low-light situations: (1) a blurred image taken with low shutter speed and low ISO noise; and (2) a noisy image captured with high shutter speed and high ISO noise. Slicing the blurred image into patches, we extend the Gaussian mixture model (GMM) to model the underlying intensity distribution of each patch using the corresponding patches in the noisy image. We compute patch correspondences by analyzing the optical flow between the two images. The Expectation Maximization (EM) algorithm is utilized to estimate the parameters of GMM. To preserve sharp features, we add an additional bilateral term to the objective function in the M-step. We eventually add a detail layer to the deblurred image for refinement. Extensive experiments on both synthetic and real-world data demonstrate that our method outperforms state-of-the-art techniques, in terms of robustness, visual quality, and quantitative metrics.

Figures

Tables

Table 1. TABLE I: Comparisons of average PSNR/SSIM/MSE on test images [ 4 ] which are corrupted with different types of synthetic blur. BlurType1 is linear motion blur, BlurType2 is circular motion blur, and BlurType3 to BlurType6 are complex blur mixed with multiple types of blur. See Fig. 10 for illustration.

	BlurType1	BlurType2	BlurType3
RL [40]	17.787 / 0.804 / 1244.060	18.766 / 0.822 / 1013.496	19.413 / 0.838 / 908.450
Deconvblind [24]	21.910 / 0.913 / 522.051	22.736 / 0.916 / 416.793	22.900 / 0.917 / 405.950
Whyte [52]	23.739 / 0.901 / 846.817	22.147 / 0.879 / 603.632	21.687 / 0.866 / 684.806
GFRS [27]	12.174 / 0.473 / 4305.645	13.531 / 0.550 / 3254.577	14.338 / 0.598 / 2758.658
GBBD [3]	23.823 / 0.921 / 496.064	23.128 / 0.906 / 409.134	22.687 / 0.902 / 451.751
Registration-1[19], [7]	21.180 / 0.775 / 755.719	20.565 / 0.769 / 794.853	21.297 / 0.815 / 702.828
Registration-2[1, 7]	21.667 / 0.860 / 578.877	20.178 / 0.825 / 813.343	20.257 / 0.817 / 802.075
SBD-single [64]	25.354 / 0.956 / 253.032	24.313 / 0.932 / 276.382	24.199 / 0.931 / 286.646
SBD-multi [64]	12.354 / 0.471 / 4356.241	12.728 / 0.499 / 3822.419	11.833 / 0.452 / 4521.26
GCRL+DL [58]	20.939 / 0.883 / 614.729	21.578 / 0.877 / 524.715	21.930 / 0.877 / 526.811
OGMM	27.905 / 0.966 / 144.208	25.046 / 0.944 / 230.982	25.295 / 0.947 / 225.836
OGMM+DL	27.716 / 0.966 / 141.942	25.133 / 0.945 / 226.747	25.410 / 0.948 / 220.014
	BlurType4	BlurType5	BlurType6
RL [40]	19.863 / 0.856 / 826.026	19.499 / 0.845 / 891.206	19.357 / 0.833 / 886.731
Deconvblind [24]	23.606 / 0.931 / 354.657	23.153 / 0.924 / 381.380	22.765 / 0.914 / 410.282
Whyte [52]	22.346 / 0.876 / 786.165	22.634 / 0.886 / 656.215	20.770 / 0.857 / 1011.524
GFRS [27]	14.145 / 0.594 / 2887.265	13.753 / 0.570 / 3168.338	14.526 / 0.598 / 2602.047
GBBD [3]	23.658 / 0.907 / 402.519	22.145 / 0.879 / 502.416	19.950 / 0.806 / 731.409
Registration-1[19], [7]	21.917 / 0.822 / 659.675	21.926 / 0.823 / 662.619	21.033 / 0.812 / 728.168
Registration-2[1, 7]	21.761 / 0.881 / 514.607	20.385 / 0.840 / 704.858	18.314 / 0.771 / 1122.865
SBD-single [64]	24.927 / 0.947 / 241.660	24.629 / 0.942 / 255.831	23.644 / 0.920 / 340.248
SBD-multi [64]	11.537 / 0.446 / 4748.240	11.938 / 0.470 / 4348.459	9.895 / 0.333 / 6891.413
GCRL+DL [58]	21.798 / 0.884 / 504.584	21.722 / 0.881 / 515.390	21.494 / 0.874 / 539.748
OGMM	26.277 / 0.958 / 188.181	25.937 / 0.954 / 203.715	24.724 / 0.940 / 257.072
OGMM+DL	26.453 / 0.959 / 180.985	26.087 / 0.955 / 196.959	24.811 / 0.942 / 252.102

Equations

$$p(y_k)=(1-\omega )\sum _{m=1}^M\frac{1}{M}p(y_k|x_m)+\omega \frac{1}{K},$$

$$E(X,{\mathbf{\sigma }}^2)=-\sum _{k=1}^K\log (\frac{1-\omega }{M}\sum _{m=1}^{M}p(y_k|x_m)+\omega \frac{1}{K}).$$

$$p_{mk}^{old}=\frac{e^{-\frac{||y_k-x_m|{|}^2}{2{\sigma }_m^2}}}{{\sum }_{m=1}^M{e}^{-\frac{||y_k-x_m|{|}^2}{2{\sigma }_m^2}}+\frac{\omega M(2\pi {\sigma }_m^2{)}^{\frac{d}{2}}}{(1-\omega )K}}.$$

$$Q(X,{\mathbf{\sigma }}^2)$$

$$\propto \sum _{k=1}^K\sum _{m=1}^M{p}_{mk}^{old}\frac{{\Vert y_k-x_m\Vert }^2}{2{\sigma }_m^2}+\sum _{k=1}^K\sum _{m=1}^M\frac{p_{mk}^{old}}{2}\log {\sigma }_m^2.$$

$$B(X)=\sum _{m^{\prime }\in N(m)}{\Vert x_m-x_{m^{\prime }}\Vert }^2{e}^{-\frac{d_{m^{\prime }}^2}{{2\sigma }_d^2}}{e}^{-\frac{l_{m^{\prime }}^2}{{2\sigma }_l^2}},$$

$$Q(X,{\mathbf{\sigma }}^2)=\lambda D(X,{\mathbf{\sigma }}^2)+(1-\lambda )B(X).$$

$$\frac{\partial Q(X,{\mathbf{\sigma }}^2)}{\partial x_m}=$$

$$\begin{array}{r}{x}_m^{q+1}=x_m^q+\alpha \left(\lambda \frac{\sum _{k=1}^K{p}_{mk}^{old}(y_k-x_m^q)}{\sum _{k=1}^K{p}_{mk}^{old}}-\mu \sum _{m^{\prime }\in N(m)}\frac{x_m^q-x_{m^{\prime }}^q}{e^{\frac{d_{m^{\prime }}^2}{{2\sigma }_d^2}}{e}^{\frac{l_{m^{\prime }}^2}{{2\sigma }_l^2}}}\right),\end{array}$$

$$\begin{array}{r}{{\sigma }_m^2}^{\prime }=\left(\sum _{k=1}^K{p}_{mk}^{old}\Vert x_m^{\prime }-y_k{\Vert }^2\right)/\sum _{k=1}^K{p}_{mk}^{old}.\end{array}$$

$$I(u,v)=\frac{{\sum }_{i,m}\text{mathbbm}1(po{s}_m^i=(u,v))x_m^i}{{\sum }_{i,m}\text{mathbbm}1(po{s}_m^i=(u,v))},$$

Full text

I Introduction

It is prevalent to adopt image deblurring techniques to recover quality images from blurry images. A common situation is capturing photos in dimly-lit environments (e.g., photographing moving objects in a night scene), where one can hardly get sharp and bright photos. Most likely, the taken photos are dark or blurry, depending on the camera settings and object conditions. Though a lower shutter speed can effectively increase brightness, it almost inevitably leads to blur. On the other hand, increasing the shutter speed makes the camera sensor or film exposed to limited light, resulting in dark photos. Setting a high ISO for increasing brightness is a trade-off way to obtain bright photos. Nevertheless, a higher gain setting amplifies noise which may even worsen the photo quality. Recovering quality photos from such captured blurry photos remains challenging and can hardly be resolved by the existing image deblurring techniques.

Removing blur from blurred images to achieve latent sharp images has been widely studied [51, 36]. Many approaches [26, 13] estimate the blur kernels using salient features. Such methods may fail when images are not bright enough to get sufficient features such as edges. In fact, it is difficult to model blur in real photos in some cases, because of the mix of different types of blur (i.e., complex blur). As a result, deblurring methods based on blur kernel estimation have limited performance in handling complex blur.

Compared with a single image, multiple images often show more information that can be utilized for deblurring. In this work, we attempt to exploit a pair of blurred/noisy images which can be easily obtained by changing the shutter speed and ISO settings. The noisy image contains complementary pixel information to the blurred image. Deblurring using a pair of blurred/noisy images has been sparsely treated so far [58, 2], and a close work to ours is Yuan et al. [58] in which the noisy image is used to add details lost in the deblurring process with deconvolution. Despite its convincing deblurring performance, it still suffers from two major issues: (1) the image pair is strictly constrained to be taken from the same view, which is to avoid the misalignment of pixels between two images; and (2) the blur kernel in their paper is supposed to be a single type (linear motion blur) caused by camera shake. These two constraints diminish its practical use, for example, the moving objects in row 2 of Fig. 1. As such, their technique has limited performance in the cases without meeting the above assumptions.

To overcome the above limitations, we propose a novel approach for image deblurring, with the easily obtained noisy/blurred image pair. Specifically, we first slice the blurred image (in view 1) into patches and each patch is guided to its corresponding patch in the noisy image (in view 2) by optical flow. To remove the blur and recover the latent sharp image, we extend the Gaussian Mixture Model (GMM) to model the intensity distribution of the latent sharp image, with its parameters estimated by the Expectation-Maximization (EM) algorithm. We further add a bilateral term to the objective function in the M-step of EM, to better preserve sharp features in images. We alternately update the optical flow and perform the EM algorithm for several rounds, to achieve desired deblurring results. Therefore, we refer to our method as optical flow guided GMM (OGMM). To further retain details, we add a detail layer (the denoised sharp features) back to the deblurred image, according to the pixel correspondences from the final estimated optical flow. Our method is free of blur kernels, and it can handle complex blur which is challenging for kernel estimation based techniques (e.g., space-variant blur in the first row of Fig. 1).

The main contributions of this paper are threefold. First, we propose a novel deblurring approach called optical flow guided GMM (OGMM) with a pair of blurred/noisy images as input. Second, we formulate deblurring as a parameter estimation problem, and derive an EM algorithm to optimize the involved parameters. Eventually, a bilateral term is added to the objective function of the M-step in EM to better preserve sharp features, and a detail layer is extracted to enhance the details in the deblurred image. Instead of kernel estimation or deconvolution, we make full use of the noisy image taken in a different view for deblurring.

II Related work

Blind deblurring for single image. Blind deblurring aims to accurately estimate the unknown blur kernel, based on which deconvolution is performed to recover the corresponding sharp image. There are several types of methods for blind deblurring, such as maximum a posterior (MAP) [28, 54], variational Bayes [20, 66, 52], edge prediction [23, 26] and sparse coding [65, 60]. For the MAP based methods, various strategies are presented to cope with the problem revealed by Levin et al. [30] that the failure of naïve MAP may occur because it favors no-blur explanations. Marginal distributions are considered to be maximized over all possible images [30, 20]. Image regularizations are introduced into the MAP framework [28, 55, 42] to retain salient image structures. The state-of-the-art methods for single image deblurring also depend on rich information hidden in the blur. Bai et al. [3] used reweighted graph total variation as prior to reconstruct a skeleton patch from the blurry observation. Kheradmand et al. [27] introduced a novel data adaptive objective function to handle challenging motion blur PSFs. Zhang et al. [65] established an unified framework for natural image restoration using group-based sparse representation. Yan et al. [57] proposed an image prior named Extreme Channels Prior (ECP) to help the uniform kernel estimation based on the observation that the values of bright channel pixels are likely to decrease. Hu et al. [25] utilized light streaks in the images taken in low light situations as constraints for estimating the blur model, but it only succeeds when the light streak is large. Single image blind deblurring usually encounters the bottleneck that the useful information for kernel estimation is insufficient, and can hardly output a proper blur model in real cases. Recently, deep learning based deblurring methods have also been explored [56, 38, 29]. Xu et al. [56] emphasized the use of CNN on motion kernel estimation, which performs well mostly on linear motion kernels. To further free the algorithm from the restrictions of blur kernel estimation, Nah et al. [38] and Kupyn et al. [29] proposed the networks that can handle complex blur kernels.

Multiple images deblurring. Efforts have been made to multiple images deblurring [9, 22, 58, 64, 43, 11, 67, 31]. The superiority of deblurring with multiple images lies in the complementary information provided in those images. Hee et al. [22] introduced a Gyro-Based method to cope with handshake blur caused by camera motion. Multiple blurred images can provide necessary frequency components which are missing due to blur. However, it can hardly handle object movement. Cai et al. [9] aligned multiple motion blurred frames accurately and show promising results with their tight framelet system. Li et al. [31] used two well-aligned blurred images to better estimate the blur kernel. Zhang et al. [64] estimate the latent sharp image with given multiple blurry and/or noisy images by designing a penalty function which can balance the effects of observations with varying quality and avoid local minimal. However, they assume a single type of linear motion blur or uniform blur. In fact, none of the above approaches can handle complex blur. To pursue better solutions in solving non-uniform blur, other methods [2, 68, 12, 63, 15, 16] have been proposed, in which they leveraged different forms of multiple observations, e.g., a blurred-noisy pair [2], a blurred-flash pair [68] and multiple blurred images [12, 63, 15, 16]. Arun et al. [2] exploited the gradient information on the noisy image to iteratively estimate the latent sharp image, and they seemed to show the results on the data with less intense blurs, according to the provided visual experiments. Cho et al. [12] proposed a registration based deblurring strategy, with the constraint that the dominant blur directions of the two input blurred images are required to be approximately orthogonal. In essence, [68, 12, 63, 15, 16] can hardly handle the blur arising from object movement, which are less powerful than our method.

Patch based GMM framework. Gaussian mixture model has been widely exploited in image restoration tasks [62, 47, 69, 53, 46, 44],[18, 61, 59] and point cloud processing tasks [34, 32, 33]. In [69], Gaussian mixture priors are learned from a set of natural images. By maximizing the expected patch log likelihood, an image without distortion can be reconstructed with priors. The learned patch group Gaussian mixture model (PG-GMM) by Xu et al. [53], providing dictionaries and regularization parameters, achieves a high denoising performance. The study by Zoran et al. [70] gives a comprehensive analysis that modeling natural images by GMM is effective in log likelihood scores, denoising performance and sample quality. However, GMM based learning methods commonly suffer from huge computational time and a massive dataset. We exploit GMM in a different way, which relates the patches in the noisy image with the patches in the latent image of the blurred image according to dense optical flow. In other words, we attempt to model the intensity distribution in each patch instead of learning patch based image priors to restore images.

III Method

Fig. 2 illustrates the overview of our method, which consists of two stages: deblurring and adding detail layer. The latter can be viewed as post-processing or refinement. We first adopt optical flow [19] to find the corresponding patches between the blurred image and the noisy image. We then formulate the image deblurring problem under the framework of GMM, and adopt the EM algorithm [17] to optimize the involved parameters. We further add a bilateral term to the objective function in the M-step, to prevent sharp features being smoothed out. Optical flow update and the EM algorithm are alternately called, to achieve the best deblurring results. Finally, we extract a detail layer from the noisy image and add it to the deblurred image, to better preserve the details.

III-A Patch Correspondence

The blurred image $I_{1}$ is decomposed into a set of overlapping square patches $C=\begin{Bmatrix}c_{1},...,c_{i},...,c_{P}\end{Bmatrix}$, where ${c_{i}}\in{R^{M}}$ and $M=s_{1}\times s_{1}$. $P$ is the number of the patches, and $s_{1}$ denotes the patch size in $I_{1}$, and $M$ is the number of pixels in each patch. The multiset of pixel intensities in an arbitrary patch from $I_{1}$ is denoted as $X$ ($X\in R^{M}$), and $x_{m}$ denotes the $m$-th pixel intensity in $X$. We adopt the dense optical flow (DOF) [19] to find $c_{i}$’s corresponding patch $d_{j}$ in the noisy image $I_{2}$. Note that for brighter and clearer visualization purposes, in the case of real images, brightness and contrast of $I_{2}$ are obtained by adjusting gain, bias, and gamma correction parameters. Here, patch $c_{i}$ has correspondence to patch $d_{j}$ if the two center pixels of $c_{i}$ and $d_{j}$ are connected with respect to the DOF field. The set of corresponding patches in $I_{2}$ can then be denoted as $D=\begin{Bmatrix}d_{1},...,d_{j},...,d_{P}\end{Bmatrix}$, where ${d_{j}}\in{R^{K}}$, $K=s_{2}\times s_{2}$. $s_{2}$ is the patch size in $I_{2}$. We empirically set $s_{2}>s_{1}$ to ensure sufficient training data for GMM. The pixel intensity multiset of an arbitrary $d_{j}$ is indicated as $Y$, $Y\in R^{K}$, and $y_{k}$ is the $k$-th pixel intensity in $Y$.

III-B The Probabilistic Model

Our key idea is to model the underlying distribution of pixel intensities $X$ with the noisy observation $Y$. We use $X=\{x_{m}\}$ to denote the corresponding latent pixel intensities, for slight notation misuse. We directly model the pixel intensities instead of the whole patch because the latter will suffer from significantly larger computation burden and noticeable interference due to increased size. To relate $X$ with $Y$, we assume that $y_{k}$ follows a GMM whose centroids are $\{x_{m}\}$. That is, the GMM with those centroids can generate the noisy observations. Thus, we formulate the deblurring problem under the GMM probabilistic framework. Inspired by Myronenko et al. [37], we define the probability density function of $y_{k}$ as

[TABLE]

where $p(y_{k}|x_{m})=\frac{1}{({2\pi{\sigma}_{m}^{2}})^{\frac{d}{2}}}e^{-\frac{||y_{k}-x_{m}||^{2}}{2{\sigma}_{m}^{2}}}$ denotes the $m$-th Gaussian component, and $d$ is the dimension of $x_{m}$ and $y_{k}$ ($d=1$ for gray image). An additional uniform distribution $\frac{1}{K}$ accounts for the noise, with a weight $\omega$. $\bm{\sigma}^{2}=\begin{Bmatrix}{\sigma}_{1}^{2},...,{\sigma}_{m}^{2},...,{\sigma}_{M}^{2}\end{Bmatrix}$ stands for the covariances and $\frac{1}{M}$ represents the equal membership probability for all the Gaussian components. The centroids of the GMM model are initialized by $X$. We next need to find the centroids and covariances that can best explain the distribution of $Y$.

III-C EM optimization

The centroids and covariances of the GMM can be estimated by minimizing the negative log-likelihood function [6].

[TABLE]

We use the expectation-maximization (EM) algorithm [17] to solve Eq. (2). The EM algorithm consists of two steps: E-step and M-step. E-step and M-step are alternately called for multiple iterations to achieve decent estimations.

E-step. The posterior probability $p^{old}(x_{m}|y_{k})$ is calculated based on Bayes’ theorem and the parameters in the previous iteration. $p^{old}_{mk}$ represents $p^{old}(x_{m}|y_{k})$ for simplicity.

[TABLE]

M-step. The M-step is to update the involved parameters ($X$ and $\bm{\sigma}^{2}$) based on the computed posteriors. This is equivalent to minimizing the upper bound of Eq. (2). “$new$” means calculating the posterior probability with the parameters to be estimated in the current iteration.

[TABLE]

III-D Bilateral Term

Eq. (4) can be treated as a data term, which in this work is to numerically approximate $Y$ with $X$. However, this data term only takes the pixel intensity distribution into account, without considering the spatial information. As illustrated in Fig. 3(a), sharp edges would become coarse (e.g., discontinuity) in the iteration of EM. To overcome this problem, we propose to add a bilateral term to the objective function in M-step. Inspired by the bilateral filter [48], we define the bilateral term $B$ as

[TABLE]

where $m^{\prime}\in N(m)$ denotes a neighbour pixel with its intensity equals to $x_{m^{\prime}}$. $d_{m^{\prime}}$ and $l_{m^{\prime}}$ are the spatial distance and the difference of intensity value between the neighbour pixel $m^{\prime}$ and the center pixel $m$, respectively. $\sigma_{d}$ and $\sigma_{l}$ are constants to control the degree of smoothness.

Redefining Eq. (4) as $D(X,\bm{\sigma}^{2})$ and weighing it with $\lambda$, the final objective function can be written as

[TABLE]

We next need to minimize Eq. (6), to solve the involved parameters.

III-E Minimization

In this section we explain how to solve the optimum solutions of $x_{m}$ and ${\sigma}_{m}^{2}$. We first take the partial derivation of Eq. (6) with respect to $x_{m}$,

[TABLE]

Given the mathematical difficulty to directly equate Eq. (7) to zero and solve $x_{m}$, due to the $x_{m}$ in $l_{m^{\prime}}^{2}$, we apply gradient descent to approximate the optimum $x_{m}$ instead. Gradient descent shares the huge advantage in its efficiency to iteratively update the value of $x_{m}$ with the manual step value. In each gradient descent step $x_{m}$ is updated as

[TABLE]

where $\mu=2\frac{{\sigma}_{m}^{2}(1-\lambda)}{\sum_{k=1}^{K}p^{old}_{mk}}$, $q+1$ and $q$ denote the $(q+1)$-th and $q$-th gradient descent iteration respectively. $\alpha$ represents the gradient descent step, which is set to 0.1 in our experiments. Eq. (8) is obtained by empirically scaling the gradient of Eq. (7) by ${{\sigma}_{m}^{2}/\sum_{k=1}^{K}p^{old}_{mk}}$ for the convenience of better controlling the gradient descent. $\mu$ plays a role in adjusting the proportion of the bilateral term. In general, we take $\mu\in[0.1,1.0]$ as a controllable parameter for simplicity. Notice that $p^{old}_{mk}$ is not updated in the gradient descent iteration for lowering computational burden.

$x_{m}$ is updated to $x_{m}^{\prime}$ after the gradient descent meets the termination criterion (in the experiment, the iteration stops if the iteration number reaches 50). We take the partial derivative of Eq. (6) with respect to ${\sigma}_{m}^{2}$. By solving ${\partial Q}/{\partial{\sigma}_{m}^{2}}=0$, ${\sigma}_{m}^{2}$ is updated as

[TABLE]

Notice that the step size for slicing $I_{1}$ into patches should be small so that a certain pixel can be updated in different GMM models due to overlapped patches. The final output value of a certain pixel $I_{u,v}$ is calculated by simply averaging all the updated values located at $(u,v)$,

[TABLE]

where $pos_{m}^{i}$ denotes the position of pixel $m$ in the $i$-th patch, $x_{m}^{i}$ denotes the pixel intensity of $m$ in the $i$-th patch. $\mathbbm{1}$ is an indicator function. See Fig. 4 for an example.

III-F Optical Flow Update

Blur hinders accurate estimation of optical flow, which can possibly lead to inaccurate matches in finding patch correspondences. To mitigate this issue, we alternate optical flow and the EM algorithm for multiple iterations. $I_{1}^{t}$ denotes the deblurred result in the $t$-th iteration ($T$ times in total), and is used to compute the optical flow in the $(t+1)$-th iteration. Updating optical flow increases the confidence of the patch correspondences.

III-G Detail Layer

We extract the sharp features from $I_{2}$ and add it back to ${I_{1}^{T}}$ to further preserve the details. A similar idea has been used in [58]. Since the noise in $I_{2}$ can negatively affect the quality of the detail layer, we apply the bilateral filter [48] to $I_{2}$ at first. We then obtain a mask $I_{m}$ by applying the Laplacian filter [8] to $I_{2}$, to select the retained details. Since the $I_{2}$ and $I_{1}^{T}$ are in different views, we use the DOF filed between them to find the spatial correspondence (see Sec. III-A). We can observe from Fig. 5 that the details are better recovered by adding the detail layer.

The algorithm of adding details is listed in Alg. 1. The proposed deblurring algorithm is summarized in Alg. 2.

III-H Key Parameters, Time Complexity and Convergence

To gain insights into the impact of parameters on OGMM, we analyze the influence of the key parameters (i.e., patch size in $I_{1}$: $s_{1}$, patch size in $I_{2}$: $s_{2}$, noise weight: $\omega$) on the deblurring results (Fig. 6). We also give a brief discussion on time complexity.

We first discuss the effect of patch size (Fig. 6(b)$\sim$(g)). $s_{1}$ and $s_{2}$ directly relate to $M$ and $K$, respectively, as mentioned in Sec. III-A. $s_{2}$ is required to be larger than $s_{1}$ to ensure a sufficient number of pixels in the corresponding noisy patch for running OGMM. Specifically, an equal patch size ($s_{1}=s_{2}$) or a larger $s_{1}$ may weaken the ability of our method to capture a decent intensity distribution. The low PSNR in Fig. 6(k) accounts for a less decent outcome for the equivalent patch size. It can be observed from Fig. 6(b)$\sim$(g) that the increase of $s_{1}$/$M$ and $s_{2}$/$K$ leads to worse deblurred results. This is mainly because the larger $s_{2}$/$K$ is, the less relevant information in the noisy patch which corresponds to the blurred patch is involved. In addition, a larger $s_{1}$/$M$ contains more blurred pixels. If the estimated optical flow is not sufficiently accurate, it will lead to unnatural results. Based on these observations, we empirically set $s_{1}=3,s_{2}=5$ as it achieves decent deblurring results in general. As for $\omega$, which accounts for “outliers”, we empirically found that $\omega=0.02$ is a good choice in terms of both deblurring and preserving details. A greater $\omega$ will degrade the performance, as shown in Fig. 6(h)$\sim$(j).

Similar to other EM-based algorithms, such as [53, 35], the complexity of our algorithm under EM framework mainly depends on the termination criterion and the size of data (patch size). Fig. 8 shows the changes of runtime with respect to the patch size (Fig. 6(b)$\sim$(g)) and the number of iterations (Fig. 6(b)). We observe that the runtime has an exponential-like relationship with the patch size and it is linearly proportional to the number of iterations. In addition, the curves of Eq. (2) with respect to the number of EM iterations with different parameter settings are shown in Fig. 9. It can be seen from Fig. 9(a,b,c) that the energy of Eq. (2) declines drastically in the first few iterations, and tend to be steady (i.e., converged) with increasing iterations. In practice, we empirically set the EM iteration number to 1 to 5, depending on the size of the image. In each iteration, 50 rounds of gradient descent are called. Fig. 7 shows an example of intermediate deblurred results produced at different gradient descent iterations in each round of EM update. It can be seen that the first EM update contributes the most to removing the blur ((Fig. 7(b)$\sim$(d))), comparing to the following updates (Fig. 7(e)$\sim$(j)). Also, a greater $\omega$ would lead to a slower convergence (Fig. 9(e,f)).

IV Experimental Results

We evaluate our approach on both synthetic data and real-world data. Quantitative comparisons on synthetic data with ground truth are also carried out. Besides traditional methods [40, 24, 52, 27, 3, 64, 58], we also evaluate the performance of our model by comparing it with the state-of-the-art deep learning based deblurring methods [38, 29]. Additionally, since our method depends on optical flow, we perform an analysis for it.

IV-A Synthetic Data

We first assess the performance of some current deblurring methods and our approach on ten image pairs from the publicly available dataset [4]. The dataset consists of multiple pairs of images taken from two different views in various scenes. To demonstrate the robustness of our method to different blur models, we synthetically generate six types of blur: (1) linear motion blur, (2) circular motion blur, (3) the mixture of circular motion blur, linear motion blur and Gaussian blur, (4) the mixture of two types of linear motion blur and circular motion blur, (5) the mixture of circular motion blur, zoom motion blur and two types of linear motion blur, and (6) the mixture of two types of linear motion blur and circular motion blur. The visualization of each type of blur is shown in Fig. 10. The first image in each pair is blurred with these six types of blur, respectively. Gaussian noise is added to the second image to generate the noisy image.

We compare our approach with the deblurring methods [64, 58] which can also take a pair of such images as input. To our knowledge, deblurring using a pair of blurred/noisy images has been sparsely treated so far, and the method [58] is highly close to ours. We also compare our method with five single image deblurring methods [27, 3],[52, 40, 24], including two baseline methods [40, 24]. Note that the results of the single image blind deblurring methods are only reference results (produced with the available programs provided by authors), and they do not take advantage of the second image as input. In addition, we include the comparison against the pipeline of denoising and further alignment strategies [19, 7, 1] which can alternatively generate results from the view of the blurred image (see Sec. IV-B for more information).

As suggested by previous works [49, 50], we compute three metrics (in an average sense) for quantitative comparisons: peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and mean square error (MSE). Tab. I displays the PSNR, SSIM and MSE, which are calculated between the deblurred images and the corresponding ground-truth image. We can see from Tab. I that our approach is more accurate than the two baseline methods [40, 24] and the state-of-the-art techniques [64, 58, 52],[27, 3]. Fig. 11 to Fig. 14 show the deblurring results under the mixup of different types of blur, by different deblurring methods. Notice that the close-up views focus on the blurred areas that are mixed by at least two kinds of blur. We mainly discuss the results by [64, 52, 58],[27, 3] which are typically superior to [40, 24]. Fig. 11(h,i), Fig. 12(h,i), Fig. 13(h,i) and Fig. 14(h,i) respectively show two types of registration results (Registration-1 and Registration-2) between the blurred images (Fig. 11(a), Fig. 12(a), Fig. 13(a) and Fig. 14(a)) and the denoised versions of the noisy images (Fig. 11(b), Fig. 12(b), Fig. 13(b) and Fig. 14(b)) with [7], using dense optical flow (DOF) [19] and homography [1].

Single image [27, 3]. The method of [27] is a general kernel similarity-based non-blind single image deblurring approach which states that nonlinear motion blur can be addressed. Its deblurring results, presented in Fig. 11(f), Fig. 12(f), Fig. 13(f), and Fig. 14(f), contain a large amount of ringing artifacts, which is due to the difficulty in correctly estimating its kernel similarity matrix for highly complex blur kernels. A similar issue occurs in the method of [3], which is a blind deblurring approach designed to deal with uniform blur via a combined regularization using re-weighted graph TV priors. Obviously, the mixup of different types of blurs is beyond its capacity. Tab. I reports the involved metric evaluation.

Single/Multiple images [64]. The method [64] can handle both single image and multiple images, and the authors suggest that multiple input images can estimate the blur kernel more accurately. We thus show two versions of this method for comparison. Fig. 11(j), Fig. 12(j), Fig. 13(j), and Fig. 14(j) show the version of single image deblurring results of method [64]. Mixup of the space-variant and the space-invariant blur hinders the algorithm from estimating correct blur kernel, which is directly reflected by the blurry result and low PSNR values. This also accounts for that the single image deblurring version of [64] performs well on the linear motion blur case, but fails to deal with the remaining five types of non-uniform blur. For the version with multiple input images, despite the fact that the method [64] puts emphasis on automatically distinguishing blurred images from noisy images, it tends to mistake the noisy image for the blurred image and conduct deblurring to the noisy image (Fig. 11(k), Fig. 12(k), Fig. 13(k), and Fig. 14(k)). This is because the proposed coupled penalty function in [64] judges that the intense blur leads to a higher degradation of the blurred image than the noisy image, thus unwillingly treating the noisy image as the dominant image.

Non-uniform blur [52]. The method of [52] stresses its advantage in handling non-uniform blur. Comparing to method [64], it produces clearer results and higher PSNR values. However, the mixup of blurs can cause obvious discontinuities, especially around the edges of dotted rectangles shown in Fig. 10, which increases the difficulty in estimating the blur kernel. For example, the close-up view in Fig. 13(e) shows the deblurring results close to the border of two different types of blurs. It is obvious that the border still clearly exists after deblurring by method [52], which confirms the limitation.

Blurred/noisy images pairs [58]. The method of [58] utilizes paired blurred/noisy images for deblurring, requiring the same view for both images. As can be observed from Fig. 11(l), the result of [58] mixes both appearances of the blurred and the noisy images since the noisy image dominates the final result, which makes the deblurring result appears quite similar to the noisy image in Fig. 11(l). This may further result in “ghost area” when the difference of the capturing view angle gets larger (Fig. 15(l) and Fig. 16(l)). As a result, it has low PSNR and SSIM values, reflected in Tab. I.

Our method, in contrast, requiring no further adjustment or extra kernel modeling with respect to the blur types and the border-near areas, is able to generate more visually pleasing results with higher PSNR values, which shows robustness against complex blur. Also, it is worth pointing out that the final result can be further improved by adding a detail layer onto the output of OGMM, as shown in Fig. 11(n), Fig. 12(n), Fig. 13(n), and Fig. 14(n).

IV-B Real-World Data

We test our approach on various kinds of blurred/noisy image pairs which are captured in low light environments using an off-the-shelf camera. Also, we compare our method with the state-of-the-art techniques [64, 58, 29].

We adopt the following procedure to take a real-world photo pair. First, we set a low ISO and a low shutter speed to obtain the blurred image. In the process of capturing, we add a camera shake, or move the object on purpose to produce stronger blur. Secondly, we use a high ISO and a high shutter speed to obtain the noisy image. Different from the synthetic data, the captured noisy images are too dark to use directly. Before deblurring, the noisy image is enhanced by synchronizing its brightness with the blurred image. The enhancement is achieved via gain/bias change and gamma correction, which also amplifies noise.

Fig. 15 to Fig. 17 exhibit visual comparisons on real-world data. The blur kernels estimated by [64], using single or multiple images, have difficulty in recovering the sharp image. The method [58] requires the same capturing view for the blurred/noisy image pair, which limits their applicability. As presented in the close-up view, it can be easily observed that heavy misalignment occurs when adding their generated detail layer back. The result by our method, without the need of kernel estimation, enjoys significantly better visual quality than those by the state-of-the-art methods [64, 58]. Also, in the case of relatively large view difference, our method bridges two images from different view angles and can correctly correspond the patches in the blurred and noisy images. The deblurred result has the same view with the blurred input.

**Discussion on alternative denoising and alignment. ** Applying denoising and further alignment algorithms on the noisy image of the pair is an alternative for achieving a clear image corresponding to the blurred image. In Fig. 18 and Fig. 19, we show some visual results for some well-known denoising methods [7, 10, 14, 48, 45, 21], and registration/alignment techniques [41, 19]. As can be observed from Fig. 18(g) and Fig. 19(g), the method [7] induces “mosaics” around the character edges. In Fig. 18(b,c,f) and Fig. 19(b,c), methods [10, 14, 48] suffer from the same issue that the fine structures near the edges are smoothed out, which is one potential phenomenon of traditional denoising methods. The methods [45, 21] are two state-of-the-art deep learning based denoising approaches. The method [45] builds dense connections among several types of memory blocks. The method [21] introduced a sub-network with asymmetric and total variation (TV) losses to specifically deal with real-world situations such as noise arising from in-camera processing. It can be clearly observed in Fig. 18(d) and Fig. 18(e) that they perform well on the areas with plain textures; however, noise around the context areas remains unresolved (please enlarge the corresponding areas for clear observations). In addition, both methods fail to handle intense real-world noise (Fig. 19(d,e)).

As shown in Fig. 18(h) (green box) and Fig. 19(h), in our method, denoising the noisy input inevitably leads to over-smoothness. Moreover, the view angle of the denoised image needs to be aligned with the blurred input. Fig. 18(i,j) and Fig. 19(i,j) are the results aligned by the correspondence provided by optical flow and homography [1], respectively. Obviously, these final “deblurred” results involve defects, such as distortion (Fig. 18(j) and Fig. 19(j)), misalignments (Fig. 18(i) and Fig. 19(i)) and remaining noise (Fig. 19(i,j)). This is because the two input blurred and noisy images have limited quality and cannot provide sufficiently accurate correspondence estimation for alignment.

IV-C Comparison with Deep Learning based Methods

We also compare our approach with two major deep learning based deblurring methods [38, 29] which claimed their capability in dealing with complex blur with suggested parameters and trained models. We show the comparison results in Fig. 20.

Nah et al. developed an end-to-end, multi-scale CNN for recovering sharp images without blur kernel estimation [38]. Yet, its results in Fig. 20(c) seem to be less visually pleasant, since blur is only partially removed. The method of [29] resorts to a conditional adversarial network which is optimized using a multi-component loss function to free kernel estimation. As shown in Fig. 20(d), it can hardly remove the strong blur with a single blurred image as the input, in both cases of synthetic complex blur and real-world blur.

IV-D Analysis on Guiding Optical Flow

This subsection reports qualitative and quantitative analysis on guiding optical flow. In our implementation, we simply adopt Farneback’s original dense optical flow [19], which performs well due to small change of viewing angle between the input image pair.

Optical flow between the deblurred result and the noisy image is supposed to be refined iteratively. Fig. 21 shows an example of the update of optical flow along with the deblurring in Fig. 2. In Fig. 21(f), the ground-truth flow between the original Fig. 21(a) without blur and Fig. 21(b) without noise is taken as an evaluation basis. Fig. 21(g)$\sim$(j) show the optical flow calculated between the noisy image and the deblurred result with respect to different iteration times. As can be observed, with the increase of iteration times, the EM optimization iteratively removes the blur, and the optical flow shows clear improvement. Moreover, we show quantitative results in terms of the average absolute flow endpoint error (AEE) [39] and the average angular error (AAE) [5], with respect to the ground-truth flow. Both the errors decrease along with the increase of the iteration numbers. It demonstrates that optical flow acts as a positive role in deblurring, and in return the deblurred image can also improve optical flow.

V Conclusion

We proposed a novel, robust image deblurring method with the use of a pair of blurred/noisy images. Our approach first builds patch correspondences between the blurred and noisy images, and then relates the latent pixel intensities with the noisy pixel intensities under the GMM framework. We introduced a bilateral term for better features preservation. To refine the deblurred result, we extract and add a detail layer to it. Our approach is free of blur kernel estimation and robust to various types of blur. Extensive experiments over the synthetic and real-world data demonstrate that our method outperforms state-of-the-art techniques, in terms of both visual quality and quantity.

The major limitation of our method is its dependency on optical flow. If the motion gap between the two images is too large, the accuracy of optical flow deteriorates. As a result, this would alter object appearance or reshape some sharp features, as illustrated in Fig. 22. In the future, we would like to exploit more useful and effective relationship among patches to address the issue of undesired optical flow.