Synthesizing a 4D Spatio-Angular Consistent Light Field from a Single Image

TL;DR

This paper introduces a novel end-to-end deep learning approach that synthesizes dense 4D light fields from a single image using appearance flow, achieving improved geometric consistency without relying on physical models.

Contribution

The study presents a new geometric representation and loss functions for single-image light field synthesis, eliminating the need for depth maps or secondary networks.

Findings

Outperforms previous models in quality and consistency

Successfully generalizes to arbitrary scenes

Enables applications like depth estimation and refocusing

Abstract

Synthesizing a densely sampled light field from a single image is highly beneficial for many applications. The conventional method reconstructs a depth map and relies on physical-based rendering and a secondary network to improve the synthesized novel views. Simple pixel-based loss also limits the network by making it rely on pixel intensity cue rather than geometric reasoning. In this study, we show that a different geometric representation, namely, appearance flow, can be used to synthesize a light field from a single image robustly and directly. A single end-to-end deep neural network that does not require a physical-based approach nor a post-processing subnetwork is proposed. Two novel loss functions based on known light field domain knowledge are presented to enable the network to preserve the spatio-angular consistency between sub-aperture images effectively. Experimental results show that the proposed model successfully synthesizes dense light fields and qualitatively and quantitatively outperforms the previous model . The method can be generalized to arbitrary scenes, rather than focusing on a particular class of object. The synthesized light field can be used for various applications, such as depth estimation and refocusing.