Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications

TL;DR

This paper introduces a CNN-based light field reconstruction method using a novel blur-restoration-deblur framework on EPIs, achieving high-quality results and enabling extended applications like depth enhancement and large disparity rendering.

Contribution

The paper proposes a new CNN framework with a blur-restoration-deblur process for light field reconstruction from sparse views, effectively suppressing ghosting and handling large disparities.

Findings

High-quality light field reconstruction demonstrated on various datasets.

Robustness surpasses state-of-the-art algorithms.

Extended applications include depth enhancement and unstructured input interpolation.

Abstract

In this paper, a novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We indicate that the reconstruction can be efficiently modeled as angular restoration on an epipolar plane image (EPI). The main problem in direct reconstruction on the EPI involves an information asymmetry between the spatial and angular dimensions, where the detailed portion in the angular dimensions is damaged by undersampling. Directly upsampling or super-resolving the light field in the angular dimensions causes ghosting effects. To suppress these ghosting effects, we contribute a novel "blur-restoration-deblur" framework. First, the "blur" step is applied to extract the low-frequency components of the light field in the spatial dimensions by convolving each EPI slice with a selected blur kernel. Then, the "restoration" step is implemented by a CNN, which is trained to restore the angular details of the EPI. Finally, we use a non-blind "deblur" operation to recover the spatial high frequencies suppressed by the EPI blur. We evaluate our approach on several datasets, including synthetic scenes, real-world scenes and challenging microscope light field data. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms. We further show extended applications, including depth enhancement and interpolation for unstructured input. More importantly, a novel rendering approach is presented by combining the proposed framework and depth information to handle large disparities.