Learning-Based View Synthesis for Light Field Cameras

TL;DR

This paper presents a machine learning method using neural networks to synthesize new views in light field cameras, improving image quality and potentially reducing the need for high angular resolution in consumer devices.

Contribution

A novel learning-based approach utilizing neural networks for view synthesis from sparse light field data, enhancing image quality over existing methods.

Findings

Synthesizes high-quality images surpassing state-of-the-art techniques

Uses only four corner views for view synthesis

Potentially reduces the angular resolution requirement in light field cameras

Abstract

With the introduction of consumer light field cameras, light field imaging has recently become widespread. However, there is an inherent trade-off between the angular and spatial resolution, and thus, these cameras often sparsely sample in either spatial or angular domain. In this paper, we use machine learning to mitigate this trade-off. Specifically, we propose a novel learning-based approach to synthesize new views from a sparse set of input views. We build upon existing view synthesis techniques and break down the process into disparity and color estimation components. We use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. We show the performance of our approach using only four corner sub-aperture views from the light fields captured by the Lytro Illum camera. Experimental results show that our approach synthesizes high-quality images that are superior to the state-of-the-art techniques on a variety of challenging real-world scenes. We believe our method could potentially decrease the required angular resolution of consumer light field cameras, which allows their spatial resolution to increase.