Joint Spatial and Angular Super-Resolution from a Single Image

TL;DR

This paper introduces an end-to-end deep neural network that jointly performs spatial and angular super-resolution from a single image, surpassing previous methods in quality and generalization for light field synthesis.

Contribution

A novel single deep learning framework with domain-knowledge-based loss functions for joint light field super-resolution without physical rendering.

Findings

Outperforms state-of-the-art in quality metrics

Successfully synthesizes dense high-resolution light fields

Generalizes across diverse scenes

Abstract

Synthesizing a densely sampled light field from a single image is highly beneficial for many applications. Moreover, jointly solving both angular and spatial super-resolution problem also introduces new possibilities in light field imaging. The conventional method relies on physical-based rendering and a secondary network to solve the angular super-resolution problem. In addition, pixel-based loss limits the network capability to infer scene geometry globally. In this paper, we show that both super-resolution problems can be solved jointly from a single image by proposing a single end-to-end deep neural network that does not require a physical-based approach. Two novel loss functions based on known light field domain knowledge are proposed to enable the network to preserve the spatio-angular consistency between sub-aperture images. Experimental results show that the proposed model successfully synthesizes dense high resolution light field and it outperforms the state-of-the-art method in both quantitative and qualitative criteria. The method can be generalized to arbitrary scenes, rather than focusing on a particular subject. The synthesized light field can be used for various applications, such as depth estimation and refocusing.