Planar Geometry and Image Recovery from Motion-Blur

TL;DR

This paper introduces a novel method for recovering 3D scene geometry and latent images from a single motion-blurred image of a multi-planar scene, addressing depth-dependent blur effects.

Contribution

It presents the first approach to estimate scene normals, number of planes, and camera motion from a single motion-blurred image of a 3D scene with arbitrary orientations.

Findings

Achieves state-of-the-art results on synthetic data

Successfully recovers scene geometry and latent image

Handles arbitrary plane orientations in 3D scenes

Abstract

Existing works on motion deblurring either ignore the effects of depth-dependent blur or work with the assumption of a multi-layered scene wherein each layer is modeled in the form of fronto-parallel plane. In this work, we consider the case of 3D scenes with piecewise planar structure i.e., a scene that can be modeled as a combination of multiple planes with arbitrary orientations. We first propose an approach for estimation of normal of a planar scene from a single motion blurred observation. We then develop an algorithm for automatic recovery of number of planes, the parameters corresponding to each plane, and camera motion from a single motion blurred image of a multiplanar 3D scene. Finally, we propose a first-of-its-kind approach to recover the planar geometry and latent image of the scene by adopting an alternating minimization framework built on our findings. Experiments on synthetic and real data reveal that our proposed method achieves state-of-the-art results.

Figures

Tables

Table 1. Table 1: Quantitative Comparison of deblurring using our method with other state-of-the-art blind deblurring algorithms on synthetically blurred dataset.

Method	[51]	[59]	[52]	[43]	Ours
PSNR(dB)	25.49	25.23	26.02	27.25	29.12
SSIM	0.7200	0.7573	0.7783	0.8346	0.9068

Equations

$$\mathbf{R}_{p}=\left[{\begin{array}[]{ccc}cos(\theta_{Z_{p}})&sin(\theta_{Z_{p}})&0\\ -sin(\theta_{Z_{p}})&cos(\theta_{Z_{p}})&0\\ 0&0&1\\ \end{array}}\right]$$

$$\mathbf{H}_{p}=\left[{\begin{array}[]{ccc}1+n_{X}\frac{t_{X_{p}}}{d}&\theta_{Z_{p}}+n_{Y}\frac{t_{X_{p}}}{d}&\nu n_{Z}\frac{t_{X_{p}}}{d}\\ -\theta_{Z_{p}}+n_{X}\frac{t_{Y_{p}}}{d}&1+n_{Y}\frac{t_{Y_{p}}}{d}&\nu n_{Z}\frac{t_{Y_{p}}}{d}\\ 0&0&1\end{array}}\right]$$

$$\left[{\begin{array}[]{c}x_{p}\\ y_{p}\\ 1\end{array}}\right]=\left[{\begin{array}[]{ccc}1+n_{X}\frac{t_{X_{p}}}{d}&\theta_{Z_{p}}+n_{Y}\frac{t_{X_{p}}}{d}&\nu n_{Z}\frac{t_{X_{p}}}{d}\\ -\theta_{Z_{p}}+n_{X}\frac{t_{Y_{p}}}{d}&1+n_{Y}\frac{t_{Y_{p}}}{d}&\nu n_{Z}\frac{t_{Y_{p}}}{d}\\ 0&0&1\end{array}}\right]\left[{\begin{array}[]{c}x\\ y\\ 1\end{array}}\right]$$

$$\left[{\begin{array}[]{c}\Delta x\\ \Delta y\\ 1\end{array}}\right]=\left[{\begin{array}[]{ccc}n_{X}\dfrac{\Delta t_{X}}{d}&\Delta\theta_{Z}+n_{Y}\dfrac{\Delta t_{X}}{d}&\nu n_{Z}\dfrac{\Delta t_{X}}{d}\\ n_{X}\dfrac{\Delta t_{Y}}{d}-\Delta\theta_{Z}&n_{Y}\dfrac{\Delta t_{Y}}{d}&\nu n_{Z}\dfrac{\Delta t_{Y}}{d}\\ 0&0&1\end{array}}\right]\left[{\begin{array}[]{c}x\\ y\\ 1\end{array}}\right]$$

$$\Delta x=\left[{\begin{array}[]{r}x\ y\ 1\end{array}}\right]\left[{\begin{array}[]{c}n_{X}\frac{\Delta t_{X}}{d}\\ n_{Y}\frac{\Delta t_{X}}{d}+\Delta\theta_{Z}\\ \nu n_{Z}\frac{\Delta t_{X}}{d}\end{array}}\right]$$

$$\Delta y=\left[{\begin{array}[]{r}x\ y\ 1\end{array}}\right]\left[{\begin{array}[]{c}n_{X}\frac{\Delta t_{Y}}{d}-\Delta\theta_{Z}\\ n_{Y}\frac{\Delta t_{Y}}{d}\\ \nu n_{Z}\frac{\Delta t_{Y}}{d}\end{array}}\right].$$

$$\left[{\begin{array}[]{c}\Delta x^{1}\\ \Delta x^{2}\\ .\\ \Delta x^{i}\\ \end{array}}\right]=\left[{\begin{array}[]{ccc}x^{1}&y^{1}&1\\ x^{2}&y^{2}&1\\ .&.&.\\ x^{A}&y^{A}&1\\ \end{array}}\right]\left[{\begin{array}[]{c}a_{x}\\ b_{x}\\ c_{x}\end{array}}\right]$$

$$\left[{\begin{array}[]{c}\Delta y^{1}\\ \Delta y^{2}\\ .\\ \Delta y^{A}\\ \end{array}}\right]=\left[{\begin{array}[]{ccc}x^{1}&y^{1}&1\\ x^{2}&y^{2}&1\\ .&.&.\\ x^{A}&y^{A}&1\\ \end{array}}\right]\left[{\begin{array}[]{c}a_{y}\\ b_{y}\\ c_{y}\end{array}}\right]$$

$$n_X/(\nu n_Z)=a_x/c_x$$

$$n_Y/(\nu n_Z)=b_y/c_y$$

$$n_X:n_Y:n_Z=\nu a_x/c_x:\nu b_y/c_y:1$$

$$g(\mathbf{x})=\sum\limits_{i=1}^{N}\alpha_{i}\odot\bigg{(}\sum\limits_{p\in P}\omega(p)f(\mathbf{H}_{p,i}^{-1}(\mathbf{x}))\ \bigg{)}$$

$$k({\mathbf{x}}_j,\mathbf{u})=\sum _{p\in P}\omega (p)\delta (\mathbf{u}-({\mathbf{H}}_{p,i}{\mathbf{x}}_j-{\mathbf{x}}_j))$$

$${\text{k}}_{{\text{x}}_j}={\text{M}}_{{\text{x}}_j}\mathbf{\omega }$$

$$\text{k}=\text{M}\mathbf{\omega }$$

$${\hat{\mathbf{\omega }}}^m=\underset{\mathbf{\omega }}{\min }\parallel \text{k}-\text{M}\mathbf{\omega }{\parallel }_2^2+{\lambda }_{\mathbf{\omega }}\parallel \mathbf{\omega }{\parallel }_1,$$

$$s_i^m=\underset{s\in S}{\min }\sum _{{\text{x}}_j\in X_i}\parallel {\text{k}}_{{\text{x}}_j}-{\text{M}}_{({\text{x}}_j,s)}{\hat{\mathbf{\omega }}}^m{\parallel }_2^2\text{for }i=1,2,...,N$$

$$\text{g}=\sum _{i=1}^N{\mathbf{\Gamma }}_{{\alpha }_i}{\text{W}}_i\text{f}=\text{W}\text{f}$$

$$\hat{f}=\underset{f}{\min }\parallel \text{W}\text{f}-\text{g}{\parallel }_2^2+{\lambda }_f\parallel ▽\text{f}{\parallel }_1$$

$$C(l_{\text{p}})=DC(l_{\text{p}})+{\lambda }_l\sum _{\text{q}\in {\mathcal{N}}_{\text{p}}}S{C}_{\text{p},\text{q}}(l_{\text{p}},l_{\text{q}})$$

$$DC(l_{\text{p}}=i)=\parallel \text{g}-{\text{W}}_i\text{f}{\parallel }_2^2$$

$$S{C}_{\text{p},\text{q}}(l_{\text{p}},l_{\text{q}})=1-r^{|l_{\text{p}}-l_{\text{q}}|}$$

Full text

1 Introduction

Recovery of 3D structure from images is an extensively researched area in computer vision. Algorithms for scene geometry recovery find applications in visual servoing, video conferencing, tracking, active vision, augmented reality etc. Well-known cues for depth recovery include disparity [1], optical flow [2], texture [3, 4], shading [5], defocus blur [6], and motion blur [7, 8, 9]. While depth estimation has been of general interest, some of the works in literature target the case of inferring piecewise planar geometry (Manhattan model). This was primarily motivated by the fact that the world around us can, in many cases, be modeled as being piecewise planar. Estimating a 3D geometry in terms of planar parameters has tremendous advantages including reduction in the computational complexity and robustness to pixel-level errors in depth cues.

Many works exist in the literature that specifically addresses the task of inferring planar scene geometry from a single image. To recover the surface orientation, foreshortening of texture was used as a cue in [4] whereas [3] used local variations of spatial frequencies. The orientation of text planes was estimated using perspective geometry in [10]. The work in [11] revealed the fact that higher-order correlations in the frequency domain caused by the projection of a planar texture are proportional to the orientation of the plane. [12] proposed a method to determine the surface normal using projective geometry and spectral analysis. While all the above methods work under the assumption of clean images, there exist very few works which attempt to make use of the cues from degradations (in the form of blur) to estimate plane normal. In [13], optical blur is used as a cue to estimate the planar orientation from a single image. The works in [14], [15] utilize motion blur to infer the surface normal of the scene from a single motion blurred image, but by assuming the case of in-plane translational camera motion.

There have been few attempts to estimate the complete 3D structure of scene from a single image using learning-based approaches. The work in [16] used a Markov Random Field trained via supervised learning to infer a set of plane parameters associated with the scene. [17] proposed an approach to identify multiple distinct planes, and estimating their orientation from a single image of an outdoor urban scene by learning the relationship between appearance and structure from a large set of labeled examples.

Lately, convolutional neural networks (CNN) are being increasingly used to address the ill-posedness of single image depth estimation. They are trained on specific datasets formed with the help of multi-view images or depth sensors [18, 19, 20] to predict depth map from a single image. However, the performance of these methods degrades on general test images that are different from the labeled data available during training. Moreover, accurate depth estimation becomes a challenge in the presence of blur since the fine-level depth cues get subdued in the presence of motion blur.

Motion blurred images have attracted increased attention in research [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32], owing to the ubiquity of mobile phones and hand-held imaging devices. Recent years have witnessed significant progress in single image motion deblurring. While the standard blind deblurring algorithms such as [33, 34, 35, 36, 37] consider the motion blur to be uniform across the image, various methods have been proposed to handle blur variations due to camera rotational motion [38, 39, 40, 41, 42] and scene depth variations [43, 44]. However, none of the existing approaches address multi-planar inclined scenes.

Although the problem of blur and depth estimation are individually quite challenging, a few attempts have been made in the literature to jointly tackle the two problems. Among the existing methods on motion deblurring, the ones that come close to that of ours is [43] and [44]. The work in [43] have proposed to jointly estimate depth and non-uniform blur from a single blurred image but is designed to handle only piecewise fronto-parallel planar scenes. [44] was designed to remove the motion blur effects caused in underwater imaging by modeling it via a virtual depth map characterized using a single exponential function. While few other works on depth-aware motion deblurring such as [45, 46, 47] have also been proposed, they rely on multiple observations.

Recently, various learning based attempts have been proposed to solve the problem of removing heterogeneous blur from a single blurred image. [48] trained a CNN for predicting a probability distribution of motion blur at the patch level. To recover the latent image, [49] estimated a dense motion flow with a fully convolutional neural network. [50] used adversarial training to learn blur-invariant features to perform motion deblurring. End-to-end trainable multi-scale CNN models are proposed in [51, 52] to restore the latent image directly. Although the above methods attempt to solve the deblurring problem in more generic settings, the performance of these methods depends purely on the training data and the learning capability of the underlying networks. While the learning based models have been shown to handle a few types of heterogeneous blur, their performance on generic blurred images is not guaranteed. At the same time, the performance of some of these methods on standard datasets such as [53] reveal the fact that conventional methods still outperform the learning based approaches when it comes to specific image formation models.

In this paper, we not only extend our previous work in [14] and but also bring in many other contributions. First, we show how the approach in [14] can be modified to account for the camera motions involving rotations too. We then develop a fully-automatic first-of-its-kind approach to recover the number of planes, the parameters corresponding to each plane, and the camera motion from a single motion blurred image of a scene with multiple planes. These results are then used to pose an alternating minimization problem to recover the complete scene geometry as well as the latent image of the scene. On motion blurred images, our depth estimates are more accurate than learning based approaches [19, 20] due to the cues present in the blur-kernels and additional constraints present in the algorithm of motion deblurring. In this paper, we relax majority of constraints that were being enforced in previous works. Unlike [14] which handles the case of in-plane translational motion of the camera alone, our proposed approach for normal estimation can handle more general kinds of camera motion. In addition, we also tackle the case of multi-planar scenes and propose a novel formulation for deblurring of such scenes. Unlike [43] which requires user interaction and relies on piecewise fronto-parallel planar assumption, our approach is fully automatic and uses only a piecewise planar representation of the scene.

The key contributions of our work are summarized below

• •

This is the first work in literature to perform surface normal estimation from general motion blur present in a single image.

• •

We develop a fully-automatic algorithm to estimate the number of planes, parameters corresponding to each plane, and the camera motion from a single motion blurred image.

• •

We propose an elegant alternating minimization approach to jointly estimate the scene geometry and latent image from a single motion blurred image. Our proposed approach is able to deliver state-of-the-art results on single image depth-aware deblurring.

The remainder of this paper is organized as follows. Our proposed approach for normal estimation directly from the blur kernels is introduced in Section 2. In Section 3, as an application to our findings, we propose a potential use of the estimated normals to perform blind deblurring of a scene containing multiple inclined planes. This is followed by experimental results in Section 4.

2 Normal Estimation from Blur Kernels

This section describes our approach, wherein we employ PSFs extracted from various locations in a motion blurred image to estimate the surface normal of the underlying scene. For the case of the fronto-parallel planar scene, all the blur kernel are one and the same, since all of them are at the same depth and the camera motion contains only in-plane translations. However, for the inclined planar scene, the size of the blur kernel varies with the scene depth, indicating that the blur kernels themselves carry the cue about the surface normal of the underlying scene. This is the key observation which motivated us to formulate a technique where one can determine the surface normal using pixel-shift information contained in the PSFs from different locations in a blurred image.

In general, the homography $\mathbf{H}_{p}$ is a function of $6$ dimensional (6D) camera motion (3D rotations and 3D translations). However, recent works in [38, 53] have shown that the effect of camera motion encountered in practice can be well-approximated using in-plane rotations and translations thereby reducing the space of camera motion from 6D to 3D while not compromising on the validity of image formation model. Hence, we too adopt this approximation to reduce the ill-posedness of the associated problems that we are going to address. Thus we use the homography $\mathbf{H}_{p}$ which is parameterized by translation along $X$-axis ($t_{X_{p}}$) and $Y$-axis ($t_{Y_{p}}$), and rotations about $Z$-axis ($\theta_{Z_{p}}$). Therefore the equation for $\mathbf{R}_{p}$ can be simplified to the following form

[TABLE]

Furthermore, a recent work [41] has shown that, for typical handshakes the blur induced by in-plane rotations of the camera can be very well modeled with small $\theta_{Z}$ (i.e; cos ($\theta_{Z}$) $\cong 1$ and sin($\theta_{Z}$) $\cong\theta_{Z}$). Our proposed solution for the normal estimation tries to exploit the linearization capability of small $\theta_{Z}$ approximation model for rotational motion. Thus for the case of general camera motion, the overall homography matrix can be simplified to the following form.

[TABLE]

For the case of an inclined scene with orientation ${\bf n}=[n_{X}\ n_{Y}\ n_{Z}]^{T}$, consider a single camera pose $p$ that is involved in the formation of PSF at position $\textbf{x}=(x,y)$. The camera pose $p$ shift the intensity at pixel location $(x,y)$ to a new location $(x_{p},y_{p})$, which can be determined as

[TABLE]

Eq. (3) implies that the pixel shifts are no longer a constant, but vary as a function of the spatial coordinates $x$ and $y$. This, in turn leads to variation in the blur kernels as well.

The linearity of the relationship between pixel shifts and the surface normal can be further pronounced, if the quantity being considered is the difference in the shift caused due to two different camera positions $\mathbf{t_{p_{1}}}=[t_{X_{p_{1}}}\ t_{Y_{p_{1}}}\ 0]^{T}$ and $\mathbf{t_{p_{2}}}=[t_{X_{p_{2}}}\ t_{Y_{p_{2}}}\ 0]^{T}$ as in the following relation

[TABLE]

where $\Delta x=x_{p_{1}}-x_{p_{2}}$, $\Delta y=y_{p_{1}}-y_{p_{2}}$, $\Delta t_{X}=t_{X_{p_{1}}}-t_{X_{p_{2}}}$, $\Delta t_{Y}=t_{Y_{p_{1}}}-t_{Y_{p_{2}}}$, and $\Delta\theta_{Z}=\theta_{Z_{p_{1}}}-\theta_{Z_{p_{2}}}$. The relation in Eq. (4) can be rearranged to obtain a linear relation between the unknown n and the pixel shifts along $x$ and $y$ direction induced at a location x as

\Delta x=\left[{\begin{array}[]{r}x\ y\ 1\end{array}}\right]\left[{\begin{array}[]{c}n_{X}\frac{\Delta t_{X}}{d}\\ n_{Y}\frac{\Delta t_{X}}{d}+\Delta\theta_{Z}\\ \nu n_{Z}\frac{\Delta t_{X}}{d}\end{array}}\right]

(5)

\Delta y=\left[{\begin{array}[]{r}x\ y\ 1\end{array}}\right]\left[{\begin{array}[]{c}n_{X}\frac{\Delta t_{Y}}{d}-\Delta\theta_{Z}\\ n_{Y}\frac{\Delta t_{Y}}{d}\\ \nu n_{Z}\frac{\Delta t_{Y}}{d}\end{array}}\right].

(6)

As can be deduced from Eq. (5) and Eq. (6), unlike the case for pure in-plane translations, the PSFs induced by general camera motion will be spatially varying even for the case of fronto-parallel planar scenes. The blur kernels are no longer spatially invariant for the case of fronto-parallel scene. However, by comparing the kernels from first and second row, it can be observed that the variation induced by the translational camera motion in each blur kernel still carries information about the surface normal.

Although the presence of $\Delta\theta$ preempts the recovery of surface normal directly from the quantities in the right-most column vector of Eq. (5) alone, we can utilize the information from both Eq. (5) and Eq. (6) together to overcome this issue. Let us denote the entries in the right-most column vector in Eq. (5) and Eq. (6) as $\textbf{b}_{x}=[a_{x}\ b_{x}\ c_{x}]^{T}$ and $\textbf{b}_{y}=[a_{y}\ b_{y}\ c_{y}]^{T}$. Similar to the case of in-plane translations, we can collect pixel shifts from multiple locations $(x^{i},y^{i})$ (for $i=1,2,..,A$) in the image to form a overdetermined set of linear equations in terms of the unknowns $\textbf{b}_{x}$ and $\textbf{b}_{y}$ as follows.

\left[{\begin{array}[]{c}\Delta x^{1}\\ \Delta x^{2}\\ .\\ \Delta x^{i}\\ \end{array}}\right]=\left[{\begin{array}[]{ccc}x^{1}&y^{1}&1\\ x^{2}&y^{2}&1\\ .&.&.\\ x^{A}&y^{A}&1\\ \end{array}}\right]\left[{\begin{array}[]{c}a_{x}\\ b_{x}\\ c_{x}\end{array}}\right]

(7)

\left[{\begin{array}[]{c}\Delta y^{1}\\ \Delta y^{2}\\ .\\ \Delta y^{A}\\ \end{array}}\right]=\left[{\begin{array}[]{ccc}x^{1}&y^{1}&1\\ x^{2}&y^{2}&1\\ .&.&.\\ x^{A}&y^{A}&1\\ \end{array}}\right]\left[{\begin{array}[]{c}a_{y}\\ b_{y}\\ c_{y}\end{array}}\right]

(8)

We use the difference between the extreme points of the locally estimated PSFs to compute the quantities $\Delta x^{i}$ and $\Delta y^{i}$. By making use of multiple PSFs computed from different locations in the blurred image, we can solve for $\textbf{b}_{x}$ and $\textbf{b}_{y}$ using least squares error minimization. From the estimates of $\textbf{b}_{x}$ and $\textbf{b}_{y}$, and with the help of the relations in Eq. (5) and Eq. (6), the estimated parameters and the normals are related as

[TABLE]

[TABLE]

Hence, we can obtain the components of the surface normal (upto a scale factor ambiguity) as follows

[TABLE]

The common scale factor can be removed by enforcing unit norm constraint to yield the final estimate of normal. Note that the normal estimate obtained in this way not only handles practically occurring camera motion, but also provides a normal estimate with minimal correspondence requirements. A minimum of 3 correspondences is sufficient to obtain the normal estimate by solving Eqs. (7)-(8).

3 Multi-Planar Motion deblurring

In this section, we introduce our approach for recovery of complete scene geometry and restoration of the latent image assuming the availability of a single motion blurred image of a multi-planar scene. From a single motion blurred image, we can recover the normals corresponding to all the planes. However, to recover the latent image, we need to solve for the remaining unknown variables in the image formation model.

Consider the discrete equivalent model of blurred image formation as given by

[TABLE]

From Eq. (12) it can be observed that, for latent image estimation, we need to recover the camera motion ($\omega$), depth values ($d_{i}$ for i=1,..,N), and an accurate estimate of plane segmentation masks ($\alpha_{i}$ for i=1,..,N). We first employ the inlier blur kernels and the normal estimate obtained from RANSAC to estimate the TSF ($\omega$) and the depth parameters ($d_{i}$ for i=1,..,N). This is then followed by an alternating minimization scheme where we solve for both the latent image ($f$) and segmentation masks ($\alpha_{i}$ for i=1,..,N) to yield the final restored image.

3.1 Estimation of camera motion and depth values

To estimate TSF and depth values, we make use of the inlier PSFs and the normal estimates obtained from RANSAC. Consider a spatial location x lying on the $i^{th}$ plane of the scene. The PSF at $\textbf{x}_{j}$ can be related to TSF $\omega$ as [25]

[TABLE]

Eq. (13) relates the inlier PSFs with corresponding depth values and underlying camera motion. Although the camera motion is the same for the entire image, the effective pixel motion experienced by each scene point depends on the normal and the depth value of the corresponding plane. To solve for the TSF, we define the depth of one plane to be reference depth $d_{0}$ and solve for the scalar factor $s_{i}=\frac{d_{0}}{d_{i}}$ corresponding to all other planes [25].

The relation in Eq. (13) can be expressed in matrix-vector multiplication form as

[TABLE]

where $\textbf{M}_{\textbf{x}_{j}}$ is a motion matrix which embeds the motion of a point light source at $\textbf{x}_{j}$ with respect to the camera poses in $P$, and $\textbf{k}_{\textbf{x}_{j}}$ and $\boldsymbol{\omega}$ are the column vector forms of $k(\textbf{x}_{j})$ and $\omega$. Note that the entries of $\textbf{M}_{\textbf{x}_{j}}$ depend on the plane normal and unknown scale factor $s_{i}$ too. By aggregating such relations corresponding to all the inlier PSFs we can obtain an equation of the following form.

[TABLE]

where \textbf{k}=\left[{\begin{array}[]{ccc}\textbf{k}_{\textbf{x}_{1}}^{T}\ ..\ \textbf{k}_{\textbf{x}_{c}}^{T}\end{array}}\right]^{T} and \textbf{M}=\left[{\begin{array}[]{ccc}\textbf{M}_{\textbf{x}_{1}}^{T}\ ..\ \textbf{M}_{\textbf{x}_{c}}^{T}\end{array}}\right]^{T}. The total number of inlier PSFs obtained from RANSAC is denoted as $c$. Since the measurement matrix requires knowledge of depth values corresponding to each plane, we cannot use Eq. (15) alone to solve for the camera motion $\omega$. Hence we choose to alternatively update the camera motion $\omega$ and depth values until convergence.

**TSF refinement: ** Once the scale factors are known, we can build the matrix M in Eq. (15) and then estimate $w$ by solving the following optimization problem

[TABLE]

where $m$ denotes the iteration number. We apply an $L_{1}$ norm based sparsity prior on $\omega$ to enforce the fact that camera motion will occupy only few poses in the entire search space. The weight of the prior is controlled through the scale factor $\lambda_{\boldsymbol{\omega}}$. We solve Eq. (16) using alternating direction method of multipliers (ADMM) [54] to obtain the TSF estimate $\hat{\boldsymbol{\omega}}^{m}$ for $m^{th}$ iteration.

Scale factor refinement: To refine the scale factors, we form a set of scale factors $S$ around 1, and search for the ones which satisfy the current estimate of TSFs and the inlier PSFs corresponding to each plane. We first use the camera motion $\hat{\boldsymbol{\omega}}^{m}$ obtained from previous iteration to generate the PSFs at all the locations and all the scale factors in $S$. For $i^{th}$ plane, the kernels generated from $\hat{\boldsymbol{\omega}}^{m}$ at locations corresponding to all the inlier PSFs of that plane are compared with respective inlier PSFs to update the scale factor $s_{i}$. To update $s_{i}$ we solve the following optimization problem

[TABLE]

where $X_{i}$ refers to the set of spatial locations corresponding to all the inlier PSFs of $i^{th}$ plane.

In the first iteration, we estimate the TSF by setting all the scale factors to unity (i.e; $s_{i}=1$ $\forall i$). The TSF estimate thus obtained is then used for updating the scale factors. Using the updated scale factors, we re-estimate $w$ using (16). This refinement process of $w$ and $s_{i}$ is repeated until the convergence of all the scale factors.

3.2 Image restoration and recovery of segmentation masks

In this section, we will discuss our approach to recover the latent image by making use of the estimates obtained from previous sections. Since latent image estimation requires knowledge of segmentation masks, the problem is still ill-posed. Hence we employ an alternating minimization (AM) scheme, where we iteratively repeat both latent image estimation and segmentation mask recovery to arrive at the desired solution. Details on the two sub-problems in our AM scheme is discussed next.

Latent image estimation: The relation in Eq. (12) can be expressed in a matrix-vector multiplication form as follows

[TABLE]

where g and f are the lexicographically ordered form of $g$ and $f$ ,respectively. The matrix $\textbf{W}_{i}$ which embeds the pixel motion corresponding to $i^{th}$ plane is built according to the camera motion $\omega$ and the parameters of $i^{th}$ plane. $\boldsymbol{\Gamma}_{\alpha_{i}}$ is a diagonal matrix built based on the segmentation mask $\alpha_{i}$. The matrix $\textbf{W}=\sum\limits_{i=1}^{N}\boldsymbol{\Gamma}_{\alpha_{i}}\textbf{W}_{i}$ subsumes the pixel motions corresponding to all the points in the scene. From known estimates of the scene plane parameters ($n_{i}$, $d_{i}$), camera motion ($\omega$), and plane segmentation masks ($\alpha_{i}$), we estimate the latent image $f$ by solving the following form of optimization.

[TABLE]

Here, to obtain $\widehat{f}$, we apply $L_{1}$ norm based prior (weighted by the scale factor $\lambda_{f}$) to enforce natural sparsity of latent image gradients [55] and then solve the resulting optimization using ADMM [54].

Estimation of segmentation masks: We estimate segmentation masks by posing it as a multi-label MRF optimization problem where the labels indicating the pixel assignments corresponding to each plane. This optimization is then solved using graphcut [56]. For a pixel at p, we define the cost corresponding to assigning the label $l_{\textbf{p}}$ as

[TABLE]

where $DC(l_{\textbf{p}})$ is the data cost to assign the label $l_{\textbf{p}}$ to pixel p, $\mathcal{N}_{\textbf{p}}$ is a neighborhood of pixels around p, $SC_{\textbf{p},\textbf{q}}(l_{\textbf{p}},l_{\textbf{q}})$ is the smoothness cost to assign the labels $(l_{\textbf{p}},l_{\textbf{q}})$ to the adjacent pixels p, q and $\lambda_{l}$ is the scalar weight on the smoothness term. We use the following form of cost function to compute the data cost corresponding to $l_{\textbf{p}}=i$.

[TABLE]

It is straightforward to see that the above data cost enforces the label assignment to respect the image formation model in Eq. (18). The smoothness cost $SC_{\textbf{p},\textbf{q}}(l_{\textbf{p}},l_{\textbf{q}})$ has the following form.

[TABLE]

where $r$ is a scalar value. This is used to enforce the fact that adjacent pixels in the image are more likely to have identical labels, i.e; the pixels corresponding to a single plane will form a contiguous region.

We start our AM by solving for the latent image by initializing $\alpha_{i}$ corresponding to the background layer as all $1$s and other layers as all [math]s. This is then followed by alternative refinement of both mask and the latent image to yield the final restored image as well as an accurate layer segmentation map.

4 Experiments

In this section, we validate the proposed method on both synthetic and real examples. We also show quantitative and qualitative comparisons with state-of-the-art blind deblurring approaches. For normal estimation of all the scene planes, we have estimated the PSFs from overlapping patches of size $120\times 120$ with an overlap factor of $50$. To estimate the blur kernel for a selected patch we used an off-the-shelf blind motion deblurring technique in [35]. To find the extremities of blur kernels, we use the PSF end point localization approach from [9]. These PSF estimates are then used in our RANSAC [57] based approach to identify the number of planes and associated normals. In the RANSAC algorithm, the PSF estimate which induces a deviation of more than $11$ degrees in the normal estimate is treated as an outlier.

In all our experiments, the number of iterations for alternating refinement of TSF and depth values in Section 3.1 as well as the AM between the latent image estimation and segmentation mask recovery was set to $5$. To solve various optimization problems discussed in previous sections, the value of $\lambda_{\boldsymbol{\omega}}$, $\lambda_{f}$, and $r$ were set to $0.1$, $0.002$, and $0.8$, respectively. All these parameters were found empirically through experimentation. For the segmentation mask recovery using Eq. (20), we used the image obtained by applying $L_{0}$ smoothing filter [58] on the estimated latent image from Eq. (19). As observed in [43], the $L_{0}$ smoothing filter not only helps in countering the adverse effects of the small edges during depth estimation, it also helps in recovering strong gradients in the latent image which, in turn, ensure better convergence of the subsequent AM approach.

4.1 Synthetic Experiments

To generate synthetic test examples, we used the trajectories from the dataset of [53] to simulate the camera motion. Images from the data-set of [60] were used as ground truth images corresponding to different layers. Layer masks were formed by manually creating binary masks of arbitrary shapes. For all the synthetic experiments we set the focal length to be 1000 pixels.

To perform quantitative evaluation of our proposed scheme for normal and latent image estimation, we created a dataset of synthetic examples comprising of 10 blurred images corresponding to 3D scenes with single and multiple planes. We verify the performance of our normal estimation scheme by finding the angular error between the ground truth normal and estimated normal. For the performance comparison of our deblurring scheme, we have used PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Measure) values of the restored images calculated with respect to the corresponding ground truth images. These values are compared with state-of-the-art deblurring approaches, by generating their results using the implementations provided by respective authors.

Fig. 1 and Fig. 2 show synthetic examples corresponding to a $2$ and a $3$ layer scenes, respectively. In both cases, images were blurred using camera motion involving translations and rotations and the background was set to be fronto-parallel. While the foreground layer of example in Fig. 1 was blurred using ${n}=[0\ -0.3162\ 0.9701]^{T}$, we used the normals $[0.3162\ 0\ 0.9487]^{T}$ and $[-0.3162\ 0\ 0.9487]^{T}$ for the two foreground layers in Fig. 2. For scene in Fig. 1, the estimated normals using the proposed method was found to be $[-0.07\ -0.3533\ 0.8956]^{T}$ and $[-0.1\ -0.1\ 0.92]^{T}$ which amounts to an average error of $7.8$ degrees. Proceeding similarly for Fig. 2, the average angular error for the three normals was found to be $6.6$ degrees. The average angular error for our synthetic dataset is $8.15$ degrees.

Qualitative comparisons for deblurring are shown in Figs.1 and 2. It can be seen that our approach recovers scene texture faithfully, while the results of existing methods contain visible artifacts. The learning based approaches [51], [59] and [52] contain artifacts at the planar boundaries and in dense textured regions. Although undesirable, such local deviations from ground-truth are often found in results of generative models, since the outputs of these networks are not constrained to follow the image formation model. The approach of [42] leads to deblurring of only few regions since it does not model depth variations. Similar issues are found in the results of multi-planar deblurring algorithm of [43] in Fig. 1 and 2, as it does not handle inclined planes. Note that manually marked regions (belonging to each plane) were provided as input to [43]). In contrast, our method is able to automatically segment the scenes and deblur them faithfully. The superiority of our results is also reflected in the quantitative comparisons provided in Table 1.

4.2 Real Experiments

The real experiments are carried out using images captured with Xiomi Mi5 camera in the presence of general camera shake. For the purpose of comparison of deblurring, we applied the conventional non-uniform motion deblurring method of [43] and the learning based models of [51], [59], and [52] to individual images. For depth estimation, we compare with the recent learning based single image depth estimation methods of [19] and [20].

In the first example, we consider a scenario where a large billboard is present at an inclination to the camera. The scene can be modeled as a single inclined plane. By following the same procedure as outlined in the synthetic case, outlier PSFs were removed and only the authentic blur kernels were used to estimate the TSF. Note that for real examples, we do not have knowledge of the true normal. Using our algorithm, the estimated value of normal for this image is $[0.2379-0.17380.9040]^{T}$ which is visually consistent with the scene inclination. Note that our estimated depth-map appears more consistent with the scene than the results of [19, 20], since we utilize the information present in the blur-kernels and enforce a planar constraint.

Our depth-estimates concur with the scene geometry. The results of [19, 20] do describe the scene depth-variation at a very coarse-level but contain various depth-discontinuities at fine-level. The superiority of our depth segmentation can be attributed to the constraints present in our deblurring algorithm.

In terms of deblurring performance, The method of [43] is able to partially deblur some regions in the scene (due to the manually supplied depth-segmentation as input), but suffers from incomplete deblurring and ringing artifacts in inclined regions. The results of [51], [59], and [52] suffer from incomplete deblurring while introducing artifacts in textured regions. Our method leads to better deblurring results.

The next set of examples containing $3$ layered scenes are shown in Fig. 3. The intermediate results for iterative depth estimation on the 4th test image are shown in Fig. 4(b-f), Note that the three different planes are clearly distinguishable in the final iteration. Again, it can be seen that our approach recovers scene depth and texture faithfully, while the results of existing methods contain visible artifacts in inclined regions.

5 Conclusions

We formulated the underlying relationship between the surface normal of a planar scene and the induced space-variant nature of blur due to camera motion. By utilizing the correspondences among the extreme points of the PSFs, we proposed a new approach to solve for the surface normal of a planar scene. The method leads to robust normal estimation even on real images which can be conveniently plugged into existing image formation model for restoration of motion-blurred 3D scenes. Finally, we proposed a first-of-its-kind scheme to estimate orientation of multiple planes from a single motion blurred image and utilized it to deblur the image. Our proposed approach achieves state-of-the-art results for the task of single image 3D scene motion deblurring.

Refined and complete version of this work appeared in the Journal of Machine Vision and Applications 2021.