A Unified Learning Based Framework for Light Field Reconstruction from Coded Projections

TL;DR

This paper introduces a unified deep learning framework capable of reconstructing high-resolution light fields from various coded projections, significantly advancing computational imaging techniques for 3D scene capture.

Contribution

It proposes a novel three-stage neural network approach that works across multiple multiplexing schemes to reconstruct light fields from minimal coded images.

Findings

Achieves state-of-the-art reconstruction quality across all tested schemes.

Comparable results to view synthesis methods from multiple images.

Enables high-resolution light field capture with conventional cameras.

Abstract

Light field presents a rich way to represent the 3D world by capturing the spatio-angular dimensions of the visual signal. However, the popular way of capturing light field (LF) via a plenoptic camera presents spatio-angular resolution trade-off. Computational imaging techniques such as compressive light field and programmable coded aperture reconstruct full sensor resolution LF from coded projections obtained by multiplexing the incoming spatio-angular light field. Here, we present a unified learning framework that can reconstruct LF from a variety of multiplexing schemes with minimal number of coded images as input. We consider three light field capture schemes: heterodyne capture scheme with code placed near the sensor, coded aperture scheme with code at the camera aperture and finally the dual exposure scheme of capturing a focus-defocus pair where there is no explicit coding. Our algorithm consists of three stages 1) we recover the all-in-focus image from the coded image 2) we estimate the disparity maps for all the LF views from the coded image and the all-in-focus image, 3) we then render the LF by warping the all-in-focus image using disparity maps and refine it. For these three stages we propose three deep neural networks - ViewNet, DispairtyNet and RefineNet. Our reconstructions show that our learning algorithm achieves state-of-the-art results for all the three multiplexing schemes. Especially, our LF reconstructions from focus-defocus pair is comparable to other learning-based view synthesis approaches from multiple images. Thus, our work paves the way for capturing high-resolution LF (~ a megapixel) using conventional cameras such as DSLRs. Please check our supplementary materials $\href{https://docs.google.com/presentation/d/1Vr-F8ZskrSd63tvnLfJ2xmEXY6OBc1Rll3XeOAtc11I/}{online}$ to better appreciate the reconstructed light fields.

Tables

Table 1. TABLE I: Comparison of CLF reconstruction for 5 test LFs ( 5 × 5 × 320 × 500 5 5 320 500 5\times 5\times 320\times 500 ) from Kalantari et al. [ 6 ] dataset.

Approach	5 Test LFs
Dictionary [8]	32.46, 0.803
Regression [20]	33.77, 0.922
Ours CLF	35.62, 0.959

Table 2. TABLE II: Coded aperture reconstruction results on 30 test LFs ( 7 × 7 × 320 × 500 7 7 320 500 7\times 7\times 320\times 500 ) from Kalantari et al. [ 6 ] dataset

	$N=1$	$N=2$
Inagaki et al. [11]	34.71, 0.85	39.23, 0.93
Ours	37.25, 0.94	38.80, 0.96

Table 3. TABLE III: Comparison of different light field reconstruction ( 7 × 7 × 320 × 500 7 7 320 500 7\times 7\times 320\times 500 ) algorithms on test sets of Kalantari et al. [ 6 ] and Flowers dataset [ 18 ] . Except the single image approach of Srinivasan et al. [ 14 ] rest of the approaches are trained on the respective training sets. Single image approach is trained on Flowers dataset alone as it does not generalize on the Kalantari et al. dataset due to small number of LFs.

	Kalantari et al.	Flowers
Approach	TestSet (30)	TestSet (100)
Single image [18] (N=1)	35.55, 0.930	37.88, 0.935
Compressive LF (N=1)	37.29, 0.951	N/A
Coded aperture LF (N=1)	36.84, 0.938	N/A
Defocus (N=1)	36.42, 0.929	N/A
Focus-Defocus (N=2)	38.21, 0.957	38.19, 0.946
View synthesis [6] (N=4)	37.51, 0.956	38.31, 0.938

Equations

$$I_c(\mathbf{x})={\int }_{\mathbf{v}}f(\mathbf{x},\mathbf{v})L(\mathbf{x},\mathbf{v})d\mathbf{v},$$

$$I_c(\mathbf{x})={\int }_{\mathbf{v}}f(\mathbf{v})L(\mathbf{x},\mathbf{v})d\mathbf{v},$$

$$\hat{L}(\mathbf{x},0)=v(I_c(\mathbf{x})),$$

$$D(\mathbf{x},\mathbf{v})=d\big{(}\hat{L}(\mathbf{x},0),I_{c}(\mathbf{x})\big{)},$$

$$\hat{L}(\mathbf{x},q)=\hat{L}(\mathbf{x}+qD(\mathbf{x},q),0).$$

$${\mathcal{L}}_{rec}=\Vert \hat{L}(\mathbf{x},\mathbf{v})-L(\mathbf{x},\mathbf{v}){\Vert }_1.$$

$${\mathcal{L}}_{dc}=\Vert D(\mathbf{x},\mathbf{v})-D(\mathbf{x}-qD(\mathbf{x},\mathbf{v}),\mathbf{v}+q){\Vert }_1.$$

$$\sum _{i,d_i}{\mathcal{L}}_{rec}(L_i,{\hat{L}}_i)+{\lambda }_{dc}{\mathcal{L}}_{dc}(D_i)+{\lambda }_{tv}{\mathcal{L}}_{tv}(D_i),$$

Full text

††This paper has supplementary downloadable material available at http://ieeexplore.ieee.org., provided by the author. The material includes the animations of reconstructed light fields. This material is 93MB in size.

I Introduction

Light field imaging has renewed interests for 3D modeling with the availability of commercial light field cameras, such as, Lytro and Raytrix. By capturing angular rays arriving from a scene point it offers post capture control over focal plane and different perspectives of the scene. However, capturing the 4D light field on a 2D sensor presents resolution trade-off, where, usually the spatial resolution is sacrificed in order to accommodate the angular resolution.

To address this LF resolution trade-off issue, a wide variety of solutions have been proposed such as super-resolution techniques [1, 2, 3], view synthesis approaches [4, 5, 6] and computational cameras with specialized optical multiplexing hardware [7, 8, 9, 10, 11]. View synthesis approaches enhance angular resolution from a sparse set of views by synthesizing the novel views via disparity estimation. Computational imaging approaches optically multiplex the incoming spatio-angular light rays to 2D coded projections on the sensor and reconstruct light field back from these projections. Other methods work by reconstructing 4D LF from 3D focal stack [12, 13].

More recently, data driven techniques using deep networks have been proposed for addressing resolution trade-off. For example, [6] use four corner views of a $8\times 8$ light field to interpolate the intermediate views by employing a learning based view synthesis approach. However, note that this approach needs access to a sparse set of views e.g. four corner views from the light field for enhancing its angular resolution, thereby making it difficult to integrate it with regular consumer cameras. Another learning based technique [14] attempts to reconstruct LF from a single image, which is an extremely ill-posed problem. They show visually convincing results for a particular dataset of flowers. However, since single image lacks geometric cues, the approach doesn’t generalize well for other scenes.

Inspired by the success of deep networks for light field reconstruction, we leverage deep networks for reconstructing light field from coded projections. For this we consider three frameworks that broadly span the spatio-angular multiplexing schemes. The first scheme is compressive light field (CLF) imaging, where a code is placed near to the sensor [8]. The modulation along both the spatial and angular dimensions enables LF reconstruction from a single coded image. The second scheme that we consider is the coded aperture (CA) imaging with the code placed at the camera aperture [15, 16]. With modulation along angular dimension alone, this scheme usually requires multiple coded images as input [9, 10, 11]. However, we show that using our deep learning approach, we are able to reconstruct LF from a single coded image.

Relaxing the hardware requirements further, for the first time, we propose light field reconstruction from two images captured using a conventional camera by varying the camera aperture. This brings us to our third scheme of LF reconstruction from a pair of all-in-focus and defocus image, where we exploit the relative defocus cue between the two images. The main advantage of focus-defocus scheme over those of CA and CLF is that we can capture these images using just a conventional camera by varying its aperture. Note that reconstructing LF from just a single defocused image is extremely difficult. The angular modulation function here is a simple averaging operation and hence it is not as well-conditioned as others for inversion.

I-A Unified framework for light field reconstruction

We take advantage of the fact that in all the multiplexing schemes the geometric or depth cue is explicitly encoded in the coded projections. Thus, we leverage on this cue to propose a disparity based light field reconstruction algorithm, where we first estimate the depth or disparity map and then use this to reconstruct the LF. The disparity estimation step and the LF reconstruction (or warping) step are common for all the three coding schemes and hence we propose an unified framework for these steps, (see Fig. 2).

More specifically, our unified LF reconstruction framework consists of the following steps. The first step is view reconstruction, where we reconstruct the centerview of the light field from the coded projection. The second step is disparity estimation, where by using the relative information between the reconstructed centerview and the coded projection, we estimate the disparity field. The disparity field is basically disparity map at all views of the light field. Disparity field enables us to perform seamless backward warping of the centerview to all the novel view points in the light field. Once we have both the centerview and disparity field, we warp the center view using the disparity field to generate the Lambertian light field. Since warping the centerview does not account for recovering the occluded pixels, we further refine the light field using a residual learning block along the lines of [14]. For the tasks of view reconstruction and disparity estimation, we use two deep neural networks - ViewNet and DisparityNet in our pipeline (see Fig. 2). ViewNet is adapted from an earlier work of image restoration by Mao et al. [17]. DisparityNet is inspired from the disparity estimation architecture of Srinivasan et al. [18] to which we add an additional block for multiscale feature aggregation. This helps our disparity estimation in ambiguous regions and improves our LF reconstruction performance by 1dB as we show later in ablation study.

Using the proposed unified framework, we compare our LF reconstruction with state-of-the-art methods in each of the multiplexing schemes. For compressive light field reconstruction (CLF) we compare our results with dictionary learning approach of Marwah et al. [8] and direct regression approach of Nabati et al. [20]. For the case of coded aperture (CA) we compare our results with the recent learning based approach of Inagaki et al. [11] which directly reconstructs LF from coded image without any geometry estimation. Since there is no previous work in reconstructing LF from focus-defocus (FocDef) pairs we compare this with recent learning based view synthesis approaches such as the four corner views method of Kalantari et al. [6] and the single image based LF reconstruction approach of Srinivasan et al. [14]. Our experiments show that in all these three multiplexing schemes, our disparity based LF reconstruction pipeline performs either on par with or better than the state-of-the-art. For CLF, our disparity based LF reconstruction outperforms the traditional dictionary learning approach [8] and learning approach [20] which directly reconstructs LF without any intermediate disparity estimation. In case of CA, our method using just one coded input produces LF reconstructions that are on, average, 2dB better than the competing method of [11]. Our LF from focus-defocus approach is especially attractive as with no hardware modification, it produces better LF reconstructions than CLF and CA. Foc-Def also produces slightly better results than the state-of-the-art learning-based view synthesis approaches from multiple images. Finally, we demonstrate that focus-defocus enables us to reconstruct full sensor resolution LF from just two images captured using a conventional DSLR camera (see Fig. 12).

Our contributions are as follows:

• •

We propose a unified deep learning based framework for LF reconstruction from coded measurements. Our main step is the disparity estimation network DisparityNet, which we train using indirect supervision from view-synthesis.

• •

Using our reconstruction algorithm we analyze three different LF reconstruction frameworks 1) compressive light field (CLF) 2) coded aperture (CA) and 3) focus-defocus (FocDef) image pair. We believe that this is the first work to look at LF reconstruction from focus-defocus image pair.

• •

With our unified LF reconstruction pipeline we show that we are on par with the state-of-the-art techniques in all the multiplexing schemes. Our FocDef approach with no additional hardware requirement performs better than hardware intensive CLF and CA. Thus, it paves the way for capturing LFs with conventional cameras.

• •

Finally, we demonstrate that we can reconstruct high resolution light field from real focus-defocus image pair captured using a DSLR.

II Related work

Light field imaging. Light field data [21] captures the spatio-angular information, that is, light rays coming at different angles from a point in the scene. By capturing the spatio-angular information light field enables synthesis of novel views [22], post-capture control over focal plane and aperture [23]. Early works used camera arrays [22] to capture light fields which are bulky for regular use. Recently, lenset based approaches [23] for capturing LFs have been successfully adopted into consumer cameras like Lytro [24] and Raytrix [25]. Due to limited sensing resolution, capturing light fields present a trade-off between the spatial and angular resolution.

Super-resolution of light fields. Many initial approaches propose to do angular super-resolution of light fields using a suitable prior [1, 2, 3]. Zhang et al. [26] and Didyk et al. [27] propose phase based approaches to synthesize light fields from the input stereo pairs. Recently, neural network based approaches, such as Yoon et al. [28], Gu et al. [29], Wu et al. [30] train deep convolutional networks (CNN) for spatio-angular super-resolution.

Light field from coded projections. Many computational imaging techniques were proposed for light field reconstruction from coded projections of spatio-angular rays. Liang et al. [9] propose LF reconstruction from a programmable aperture camera with optimal multiplexing patterns. The number of coded images they require as input is as high as the LF angular resolution itself. Ashok et al. [7] suggest that by exploiting the spatio-angular correlations inherent in LFs, they can be reconstructed from much less measurements than implied by their angular resolution. Babacan et al. [10] use programmable aperture to do LF reconstruction from coded images with reduced number of measurements than [9]. Marwah et al. [8] realize the hardware for compressive light field camera by placing a mask near the sensor. This spatio-angular multiplexing enables them to reconstruct full sensor resolution LF from just a single coded image via LF dictionary and optimal projections. Hirsch et al. [31] propose a switchable camera with angle sensitive pixels which can capture high resolution 2D images and 4D light fields. In addition, other set of works explore reconstructing 4D light field from 3D focal stack. For this purpose, Levin et al. [12] use dimensionality gap prior and Mousnier et al. [13] use tomography based back-projection. Instead of reconstructing light field from focal stack which uses tens of images, in our third scheme, we propose to reconstruct LF from just a pair of focus-defocus images. Disparity based view synthesis for light field cameras. These approaches involve two steps for view synthesis. First step involves estimating disparity at the novel view point. Second step involves warping the available views and weighing them appropriately to finally render the novel view. Wanner et al. [4] propose a variational framework for disparity estimation for novel view synthesis to enhance LF angular resolution. Flynn et al. [32] use deep neural networks for novel view synthesis from a set of camera views with wide baseline. Kalantari et al. [6] propose a learning based view synthesis for light field cameras. They propose to do full light field ($8\times 8$) reconstruction from just four corner sub-aperture images. Inspired from the recent works of unsupervised geometry estimation [33, 34, 32], they propose to learn disparity using the view-synthesis loss. By simultaneously learning both the disparity estimation and color prediction they show visually convincing results. This is specially so around occlusion edges where earlier approaches of Wanner et al. [4] exhibit tearing artifacts. Our approach differs from these view synthesis approaches basically in the form of input for LF reconstruction. These approaches typically have multiple views as input whereas our framework, in case of CLF and CA, uses just a single coded image.

Single image based LF synthesis. Srinivasan et al. [14] attempt to reconstruct a 4D light field from a single RGB image. They propose a disparity based approach where disparity is estimated from a single image using a deep network. Using this disparity map, they warp the single image to obtain a Lambertian LF. This is followed by a refinement step to account for non-Lambertian effects like occlusions. Since single image geometry estimation is ill-posed, they train their method on specific category of scenes. In contrast, our framework exploits explicit geometric cues to synthesize the light field from the coded image and hence can generalize to a variety of scenes.

Depth from coded projections. Many works were proposed for estimating depth and all-in-focus image from coded projections such as [15, 16, 35]. Depth from focus-defocus pairs is also well studied in the depth from defocus (DfD) literature [36, 37, 38]. In our coded aperture case similar to Levin et al. [15] we also recover image and depth from a single coded capture but with the goal of reconstructing light field. In focus-defocus case, disparity estimation requires disambiguation of the depth planes in-front of focal plane with the depth planes behind it. DfD works do not account for this and typically require the focal plane to be on the nearest depth plane. The multi-aperture imaging work of Hasinoff et al. [38] proposes a greedy approach for depth disambiguation for restricted set of scenes. Our disparity estimation network on the other hand learns to do the depth disambiguation via light field reconstruction loss.

In this work, we extend upon the work of [39] which does disparity based compressive LF reconstruction. We extend it to another popular multiplexing scheme, which is LF reconstruction from a single coded aperture image, and also propose a third scheme, which is LF reconstruction from focus-defocus image pair. From the algorithm point of view, [39] predict only the centerview disparity map in the light field. Thus, for rendering light field they use forward warping which requires hole-filling in the novel view. On the other hand, we estimate the disparity field, disparity map at every view of LF, which enables seamless backward warping with no requirement for hole-filling. Also, we refine the warped light field to account for occlusion. Further we analyze the performance of the three schemes, compare them with state-of-art LF reconstruction algorithms and finally demonstrate our algorithm on real focus-defocus pairs captured from a DSLR.

III Acquiring Coded projections

As we discussed, we consider three frameworks for light field reconstruction from coded projections. Here we describe mathematically the projection of incident spatio-angular light field to a 2D coded image in detail for all these setups. For this, we follow the two plane parametrization for spatio-angular light field $L(\mathbf{x},\mathbf{v})$, where $\mathbf{x}$ and $\mathbf{v}$ are the 2D spatial co-ordinates on the sensor plane and 2D angular position on the aperture plane, respectively.

III-A Compressive light field (CLF) photography

Here, we consider the optical multiplexing proposed by [8]. This involves placing the coded mask near the sensor and in between the sensor and lens (see Fig. 1). The coded image formation is given as follows,

[TABLE]

where the incoming spatio-angular light field is modulated by the mask $f$ along both spatial and angular dimensions. Due to this, each position on the sensor plane observes a different projection of angular rays. Thus, the spatio-angular modulation provides best case of acquiring incoherent measurements. This enables for a 4D light field reconstruction from a single 2D coded projection [40, 8].

III-B Coded aperture (CA) light field photography

Coded aperture is a further simplification of multiplexing with coded mask being moved on to the aperture [15, 16]. The coded image formation is as follows:

[TABLE]

with modulation only along the angular dimension. Since the coded mask is on aperture plane now, essentially, each pixel on the sensor plane observes the same projection of angular rays. This provides certain amount of hardware relaxation than compressive LF at the cost of multiplexing being spatially invariant. Thus, coded aperture frameworks [10] accommodate for this with multiple measurements for light field reconstruction.

III-C Focus-defocus (FocDef) pair

Here we relax the explicit coded projections in the above cases and propose to use the relative defocus cue between two images introduced by aperture variation for light field reconstruction. The defocus image can be thought of as a special case of coded aperture imaging with code $f$ being uniform and the in-focus image as the pin-hole view. The aperture variation cue relaxes to a greater degree the specialized hardware necessary for light field reconstruction. Thus, paving the way for high resolution light field reconstruction with conventional cameras like DSLR.

IV Light field from coded projections

Our goal is to reconstruct a 4D light field $L(\mathbf{x},\mathbf{v})$ from its 2D coded projection $I_{c}(\mathbf{x})$. We propose a deep learning architecture to learn this mapping from the coded image to the light field. Instead of directly reconstructing light field from coded projections, we break it down into view reconstruction and disparity estimation steps. By enforcing disparity estimation as an intermediate step for LF reconstruction our approach provides an interpretable solution. The estimated disparity values directly translate to slopes of epi-polar image (EPI) lines in the reconstructed LF. Whereas, in case of direct reconstruction no such interpretation is possible. The neural network has to explicitly learn to extract the geometry which is not very efficient as shown by recent works of [32, 6, 14].

We breakdown the light field reconstruction from coded images into three stages. In the first stage we train a deep network for reconstructing the centerview of the light field $L(\mathbf{x},0)$ from the coded image $I_{c}(\mathbf{x})$:

[TABLE]

where $v$ is a deep neural network which we call as ViewNet, and $\hat{L}(\mathbf{x},0)$ is the predicted centerview. In the second stage, we train a deep network to estimate the disparity field $D(\mathbf{x},\mathbf{v})$, that is, disparity map for every spatial position and angular view $(\mathbf{x},\mathbf{v})$ in the light field. Note that if we predict the centerview disparity alone instead of the whole disparity field then we would have to employ forward warping of centerview which might leave holes in the interpolated image. We leverage the relative information between the coded image $I_{c}(\mathbf{x})$ and reconstructed centerview $\hat{L}(\mathbf{x},0)$ to estimate the disparity field:

[TABLE]

where $d$ is a deep network which we call as DisparityNet. The disparity map at a view point is defined with respect to its immediate neighbour. Note that estimating disparity maps at novel views would require the estimation of disparity values for missing pixels around occlusion edges which are not present in the centerview. Finally, in the third stage, we perform light field synthesis by backward warping the centerview $\hat{L}(\mathbf{x},0)$ according to the disparity map $D(\mathbf{x},q)$ at viewpoint $q$ giving us the warped light field $\hat{L}(\mathbf{x},q)$ at viewpoint $q$. We repeat this process for all viewpoints to obtain the complete LF.

[TABLE]

Note, since the disparity map is with respect to its immediate neighbor we need to multiply it with $q$ to get disparity with respect to the centerview. We train the disparity based light field rendering in an end to end manner by minimizing the light field reconstruction error. We use bilinear interpolation for warping and hence the entire pipeline is differentiable. Thus, DisparityNet learns the scene geometry from light field reconstruction loss without the explicit need for ground-truth disparities. The LF reconstruction loss is given by:

[TABLE]

IV-A Disparity from coded projections

In order to estimate disparity from the coded images, we propose to learn the disparity using view-synthesis loss (Eqn. 6). Although, an alternative approach would be to compute the depth map first using conventional approaches, for example, for focus-defocus pair we can use depth from defocus approaches and then synthesize the LF from that. However, this approach of separating the disparity estimation and synthesis of novel views is shown to be ineffective in earlier works of Kalantari et al. [6], Wu et al. [30] as it leads to tearing artifacts around occlusion edges. Thus, an end-to-end training for disparity estimation and view synthesis is preferable. However, learning to estimate disparity poses certain challenges which are discussed below.

IV-A1 Challenges with disparity estimation

First of all, for constant intensity regions, any disparity value is likely to reduce the reconstruction error. Hence, there is an ambiguity as to the correct disparity value for such regions. Second challenge is in estimating the correct disparity sign for scene depth planes in-front (+ve disparity) or behind (-ve disparity) the focal plane. For example, in case of focus-defocus pair, both the depth planes in front and behind the focal plane see similar defocus blur. Only the depth edges contain the cue as to the ordering of depth planes.

To address these problems, we introduce a multiscale feature aggregation block in our disparity field estimation network shown in Fig. 3. We use dilation rates in our convolutions to expand the receptive field, that is, the region in the input image that affects the filter output. Thus, dilated convolutions allow us to leverage neighbourhood context while estimating disparity values. We additionally concatenate the features from different dilation rates allowing us to leverage context around a pixel at multiple scales. This feature aggregation step also helps us in estimating the relative position with respect to the focal plane. As we will see in our results later, disparity estimation without such feature aggregation results in erroneous predictions (see Fig. 10).

In addition, for better disambiguation between positive and negative disparities with respect to the focal plane, we additionally augment our training data by shearing the light field to vary the plane of focus in the coded images. This ensures that the disparity estimation network sees a variety of relative focal plane positions while learning the parameters.

IV-A2 Disparity consistency regularization

As discussed earlier, our DisparityNet predicts disparity field which is disparity map at ever view point or sub-aperture image of the light field. Note that for the disparity maps across the views to be consistent, all the views should have same disparity value for a 3D point in the scene. To enforce such consistency among disparity maps in a disparity field $D(\mathbf{x},\mathbf{v})$, we additionally enforce photo-consistency among disparity maps. For example, disparity at views $\mathbf{v}$ and $\mathbf{v}+q$ should satisfy, $D(\mathbf{x},\mathbf{v})=D(\mathbf{x}-qD(\mathbf{x},\mathbf{v}),\mathbf{v}+q)$ which leads to the following regularization,

[TABLE]

Godard et al. [33] and Srinivasan et al. [14] show that using such geometric consistencies among the disparity maps improves the disparity estimation over using the view reconstruction loss alone.

Loss function Overall our loss function is given by,

[TABLE]

where, $\mathcal{L}_{dc}$ ensures sparsity of $D(\mathbf{x},\mathbf{v})$ along EPI lines, $\mathcal{L}_{tv}$ ensures spatial sparsity of the disparity map for each view. The term $d_{i}$ indicated in the sum refers to shearing of LF in each batch to change the focal plane in the scene resulting in variety of relative depths (see Sec. IV-A1). In addition to this loss function we also have a refinement loss proposed by Srinivasan et al. [18]. The refinement stage takes as input the warped LF, disparity field and predicts a residual LF which is added to the warped LF resulting in the refined LF which is then compared to the ground truth.

In the following section, we will discuss the individual neural networks in detail.

V A Unified Learning Framework

V-A ViewNet: View reconstruction

As discussed earlier, the first step in our light field reconstruction framework is recovering the centerview of the light field from the coded projection. View reconstruction involves recovering high frequecy details from the coded image. We pose this as an image restoration problem and use the state-of-the-art approach of [17]. Note that view reconstruction is necessary only for the cases of CLF and CA. In the case of focus-defocus pair, the all-in-focus input serves as centerview.

ViewNet consists of encoder-decoder architecture with symmetric skip connection to preserve the low-level details. We use 15 such symmetric skip connections for centerview reconstruction for both the cases of CLF and CA. We use $3\times 3$ filters in the CNN. The network is trained by minimizing the $l_{1}$ reconstruction error.

V-B DisparityNet: Disparity field estimation

Our next step involves estimating the scene geometry from the reconstructed centerview and the coded image for rendering the light field. We concatenate both the coded image and centerview as input to our disparity estimation network for predicting the disparity field, $D(\mathbf{x},\mathbf{v})$. Although the disparity could be estimated from the coded projection alone, using relative information between the coded image and centerview makes it easier. Moreover, the centerview is anyway available from the view reconstruction step as part of LF reconstruction.

DisparityNet shown in Fig. 3 estimates disparity at all the views in the light field. The network uses dilated convolutions [41] with successive dilation rates of 2, 4, 8 and 16 which gives a receptive field of size $67\times 67$ in the input image. Then, we concatenate features from different dilation rates which combines information at different scales and is followed by few more convolutional layers to output disparity maps at all the views, in this case $7\times 7$. Each convolution layer is followed by ELU activation and batch normalization. The last layer is followed by a Tanh activation layer with a scaling factor of 10, to limit the disparities between adjacent views to $[-10,10]$. The output disparity maps are used to generate the warped light field. In addition to the reconstruction loss, disparity maps are also regularized for disparity consistency as discussed in Sec. IV-A2.

VI Training

For training our framework we use the light field dataset222cseweb.ucsd.edu/ viscomp/projects/LF/papers/SIGASIA16/ by Kalantari et al. consisting of 100 LFs for training, 30 for testing. In addition, we also show our results on Flowers dataset 333github.com/pratulsrinivasan/Local_Light_Field_Synthesis by Srinivasan et al. with 3343 light field which we randomly split into 3100 for training and the rest for testing. We refer to these test LFs as Flowers testset. This data consists of macro-shots of flowers exhibiting complex occlusion patterns and weak textured regions like petals of the flowers. In both the datasets, the resolution of original LF is $14\times 14\times 375\times 560$. Since the angular views along the border exhibit artifacts we limit ourselves to angular resolution of $7\times 7$.

We randomly crop $7\times 7\times 120\times 120$ patches from the light fields for training our deep networks. We simulate the coded images from the light field and use it as input for our light field reconstruction. For warping, we use bilinear interpolation. We implemented all our models in Tensorflow and train the parameters by minimizing the overall loss function given in Eqn. 8. Each convolutional layer is followed by exponential linear unit (ELU) and batch normalization (BN) except for the last layer in each network. The weights of the different regularizers are set to $\lambda_{dc}=0.008$, $\lambda_{tv}=0.01$. We use the Adam optimizer [42] with a learning rate of $1e-4$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=1e-8$ and a batch size of $7$. We train for a maximum of $100K$ iterations. Note that in all the three schemes for light field reconstruction - CLF, CA and focus-defocus pair, we use the same set of optimization parameters.

VII Results

In all our multiplexing schemes, CLF, CA and FocDef pair, we synthesize light field of spatio-angular resolution $7\times 7\times 320\times 500$ from coded images of spatial resolution $320\times 500$. In the following section, we will discuss the light field reconstruction from coded images along with the comparisons for each multiplexing scheme. This is followed by ablation studies which show the effectiveness of using multiscale context aggregation and refinement of LF. Finally, we demonstrate real LF reconstructions with focus-defocus pairs captured from a DSLR.

VII-A Compressive Light Field Reconstrucion

For simulating the coded image in this case, we sample code from a Gaussian distribution with values clipped to $[0,1$] and perform multiplexing similar to Marwah et al. [8]444web.media.mit.edu/ gordonw/CompressiveLightFieldPhotography/ for $7\times 7$ LF. Note the code in this case is spatially varying and is of $15\times 15$ spatial resolution which we repeat to multiplex the whole image.

Figure 4 shows the results of our compressive LF reconstruction algorithm for the test scene, Flowers2 from the Kalantari et al. data. Our disparity map captures the scene geometry well by clearly delineating the flower from the background. Dictionary learning [8] does not recover the high frequency details well as can be seen from the details shown in the inset. Also, the EPI slopes are not recovered well which is essential for LF recovery thus affecting its PSNR and SSIM values. Direct regression [20] reconstructs LF directly from the coded image without intermediate disparity. Although this approach reconstructs EPI slopes better than dictionary learning it still does not recover the high frequency details as well as our approach. In contrast to dictionary learning and direct regression, our approach does LF reconstruction in two stages of view-reconstruction and disparity estimation. By delineating the LF reconstruction into these two stages, our approach recovers light field better than the other methods as indicated in Table I and the error maps in Fig. 4.

Table I shows the average LF reconstruction results on 5 test LFs from Kalantari et al. dataset. We train light field dictionaries similar to Marwah et al. using a subset of training LFs from Kalantari et al. dataset. For reconstruction using dictionaries, we solve LASSO with ADMM. For direct regression approach evaluation, we trained the architecture of Nabati et al. [20] similar to that of our approach.

VII-B Coded Aperture Light Field Reconstruction

For simulating the coded image, we sample code from a uniform distribution in $[0,1$] range. Note the code in this case is spatially invariant (see Fig. 1) and hence multiplexes only the angular dimensions of $7\times 7$ resolution. Compared to compressive light field, the coded aperture measurements are less incoherent and hence both view reconstruction and disparity estimation are challenging.

Figure 5 shows the results of our coded aperture LF reconstruction for the test scene, Flower-1 from Kalantari et al. data. We show our reconstructed centerview from the coded image along with the disparity maps. We compare our results with the learning approach of Inagaki et al. [11] which reconstructs LF from coded images directly without any disparity estimation. For comparison, we use the coded aperture masks provided by Inagaki et al. for the case of single $(N=1)$ and dual $(N=2)$ coded aperture at $8\times 8$ angular resolution and resize them to $7\times 7$. Using these masks, we train both ours and Inagaki et al. approach on Kalantari et al. dataset. Table II shows the average reconstruction results over the testset.

As we can see from the error maps shown in the figure and Table II, for the challenging case of single coded image $(N=1)$, our approach outperforms Inagaki et al. [11] by 2 dB on the testset. The red and blue insets shown in the figure detail the top-right view reconstructed along with the EPIs for $N=1$ case. We can see that Inagaki et al. reconstruction has artifacts as evident in the blue inset. Also, it incorrectly estimates the EPI slopes, see the highlighted regions on the EPI for both the insets. For $N=2$ case, both the approaches are comparable as can be seen from error maps in Figure 5 and the average test results in Table II.

VII-C Light Field from Focus-Defocus Pair

For simulating defocus image, we take the average across all the views ($7\times 7$) which amounts to an $f$ number of $f/4$. We additionally augment the data by shearing the light field. This shearing results in shifting of the focal plane in the defocus image, thus ensuring that during training, disparity estimation network sees defocus at various depths relative to the focal plane. Since we provide the centerview explicitly as all-in-focus scene, view reconstruction is not necessary in this case.

Note that one can try to reconstruct LF from a single defocused image also. But, compared to CLF and CA, this is a severely ill-posed task as a defocused image corresponds to uniform angular multiplexing of the LF. Figure 6 shows the LF reconstruction error for different multiplexing schemes. As can be seen, the LF reconstruction from just a single defocused image performs poorly compared to CLF and CA. To compensate for the loss of angular information in the defocused image, we propose to use the all-in-focus image as well for LF reconstruction. With this addition, we get our focus-defocus LF reconstruction scheme, whose performance is even better than CLF, see Fig. 6. Moreover, the addition of the in-focus image does not involve any hardware change to the conventional camera.

Since there exists no prior work with LF reconstruction from focus-defocus pairs, we compare our LF reconstruction with the recent state-of-the-art learning based LF synthesis approaches of Kalantari et al. [6] from four corner views and Srinivasan et al. [14] from a single image. Figure 7 and Table III show comparisons with these approaches. For Kalantari et al. approach we use the code and trained models provided by the authors. For Srinivasan et al. approach we use the code made online by the authors for training and testing.

Both the approaches of Kalantari et al. and Srinivasan et al. involve disparity based LF synthesis. Figure 7 compares the centerview disparity estimated from these approaches and ours with that of the disparity map obtained from full 4D LF by EPINET [19]. EPINET is a deep network operating along EPIs of light field for disparity estimation. As we can see from the disparities, single image based approach, being trained on the specific dataset of flowers, fails to generalize across all scenes. For scenes like Leaves and Flower-1 it performs well, but for others scenes such as Flower-3, Reflective (Fig. 7) and Seahorse (Fig. 1), it fails.

LF synthesis from four corner views by Kalantari et al. uses a four layer CNN for disparity estimation with explicit stereo and defocus features as input. It results in better disparity maps but due to a rather simple CNN architecture it results in texture leakage and bleeding around depth edges and misses out fine details sometimes in disparity maps. For example, for Seahorse scene (Fig. 1) and Reflective scene (Fig. 7), disparities from corner views exhibit texture leakage which are not prominent in our disparities estimated using focus-defocus input. In Fig. 8, we compare our estimated disparities with Kalantari et al., as you can see, our estimation has lesser artifacts and preserves the fine details. Our approach performs well even for scenes like Reflective (Fig. 7) where the jar handle in the front is separated from the background, which is mostly transparent.

Table III shows the average results on testsets of Kalantari et al. [6] dataset and Flowers dataset of Srinivasan et al. [18]. The average values are reported by considering views which are not given as input to any of the methods. All the approaches are trained on respective datasets except the single image approach as it does not generalize on the Kalantari et al. training set of 100 LFs. So we train this on Flowers dataset alone and evaluate on both testsets. It performs poorly on Kalantari et al. testset whereas on Flowers testset it is relatively better. Focus-defocus LF reconstruction performs as well as the view synthesis approach of Kalantari et al. [6] on both the testsets. As expected, it performs better than the single image based LF reconstruction as it uses the defocus cue for disparity estimation whereas single in-focus image does not have any such cues. In Table III, we also show the performance of CLF and CA on Kalantari et al. testset. Note that for simulating coded images for both CLF and CA, we used random masks as discussed earlier.

VII-D Additional results on the Flowers dataset

Figure 9 shows the disparity maps estimated using the focus-defocus input for Flowers dataset. Scenes in this data exhibit challenges like constant intensity regions and complex occlusion patterns. As can be seen from the disparity maps our approach captures the fine occlusion information created by thin stems and plant leaves. Also, it estimates the disparity values accurately for flower petals which have constant intensity values. This is due to our multiscale feature aggregation step in our DisparityNet. As we show in our ablation study in the next section, without that, our estimates are inaccurate for such challenging scenes. Average results of LF reconstruction are provided in Table III.

VII-E Multiscale context aggregation

Without the context aggregation block, our DisparityNet is similar to the disparity estimation architecture proposed for single image based LF reconstruction by Srinivasan et al. [18]. Note that when using Srinivasan et al. architecture we modify their input layer for our input focus-defocus images and retrain it. Figure 10 shows the disparity estimation without the multiscale feature aggregation in DisparityNet (see Sec. 3). As can seen from the figure, our disparity estimation is qualitatively better in weakly textured regions (car surface) and exhibits less RGB texture leakage (on the belt). On the testset of Kalantari et al. consisting of 30 LFs, our network without context aggregation achieves PSNR and SSIM of $37.51$ dB, $0.944$ and with context aggregation, achieves $38.21$ dB, $0.957$.

VII-F Light field refinement

Figure 11 shows the LF reconstruction results before and after the refinement stage. Warping based LF reconstruction introduces shearing artifacts around occlusion edges. Refinement stage takes as input the warped LF and the disparity field and predicts a residual LF which is then added to the warped LF. The refined LF exhibits less artifacts around occlusion edges as seen in the Fig. 11. On the Flowers test of 100 LFs, light field reconstruction from focus-defocus without and with refinement gave PSNR and SSIM of 38.31 dB, 0.944 and 38.95 dB, 0.954, respectively.

VII-G Comparison with encoder-decoder architecture

We compare our dilated convolution and multiscale context aggregation based DisparityNet with the encoder-decoder architure based approach of Vadathya et al. [39]. We show this comparison for our best performing case of focus-defocus scheme. Vadathya et al. use encoder-decoder architecture for disparity estimation with skip connections to preserve the low-level details. We use dilated convolution to enhance receptive field and perform multiscale context aggregation to resolve ambiguities in disparity estimation. On the Kalantari et al. testset of 30 LFs our network achieves 38.21dB (PSNR), 0.96 (SSIM) whereas the encoder-decoder architecture results in 37.28 dB, 0.94.

VII-H LF synthesis from a DSLR

To demonstrate real light field reconstructions, we capture in-focus and defocus pair from a DSLR (Canon 70D) by varying the aperture (f/4, f/16). Figure 12 shows the LF reconstruction ($7\times 7\times 460\times 610$) results with focus-defocus pair captured using a conventional camera. Notice the relative disparity estimation (+ve and -ve) delineating the depth planes.

VIII Discussions and Conclusion

We propose a unified learning framework for full sensor resolution light field reconstruction which can work under a variety of spatio-angular multiplexing schemes with minimal measurements. We demonstrate this on three coded schemes: compressive LF with code near the sensor (CLF), coded aperture (CA) with code on the aperture and focus-defocus pair from a conventional camera. If we consider single shot imaging, then CLF followed by CA are better posed than just a defocused image as they do explicit coding of spatio-angular rays. However, by adding an all-in-focus image (captured with narrow aperture), along with the defocused image, the reconstruction problem becomes much better posed as we can use the fine texture details from the all-focus image and depth cue from relative defocus between the two images. The focus-defocus scheme has the additional advantage of ease of capture using conventional cameras.

Using our disparity based LF reconstruction framework, we analyze these three coded imaging schemes in terms of view reconstruction and disparity estimation. Disparity estimation is learned in an unsupervised manner by using the light field reconstruction loss. As can be seen in our results, disparity maps estimated from focus-defocus pair are better than that from CLF and are similar to those obtained from full 4D light field (see Fig. 7). Focus-defocus utilizes the high frequency details from the all-in focus image and hence results in sharper disparities as compared to CLF. Focus-defocus also recovers the relative disparities for depth planes in-front and behind the focal planes very well (see +ve and -ve disparities in Fig. 12). This delineation is challenging as local cue such as relative defocus is not enough and we need to utilize global cue such as occlusion edges. The feature aggregation step in our disparity estimation network leverages on this global cue. In addition, we also have a data-augmentation step where shearing the light field changes the focal plane. This ensures that the data has variety of relative defocus w.r.t the focal plane during training.

We compared our reconstructions with both direct regression (Nabati et al. [20], Inagaki et al. [11]) and disparity based rendering from four corner views of LF (Kalantari et al. [6]) and from a single image (Srinivasan et al. [18]). Our CLF and focus-defocus approaches perform as well as the four corner view method of Kalantari et al and outperform direct regression and single image based approaches. Moreover, our approach is much faster than that of Kalantari et al. In the four view approach the disparities are estimated independently for each view, which is computationally inefficient, whereas in our approach we compute disparities for all the views jointly.

Our learning algorithm, thus, paves the way for full sensor resolution light field reconstruction from either a single coded image with specialized hardware (CLF and CA) or with two images (FocDef) from a conventional camera.

Acknowledgment

This work is supported by Qualcomm Innovation Fellowship (QInF) India 2016 and 2017.