4D Temporally Coherent Light-field Video

TL;DR

This paper introduces a novel method for creating 4D temporally coherent light-field videos from sparse camera arrays, enabling dynamic scene representation with improved temporal consistency and accuracy over existing approaches.

Contribution

It presents the first approach to extract spatio-temporally coherent light-field videos using scene flow from sparse camera arrays, addressing challenges in data volume and temporal coherence.

Findings

Significant improvement in temporal coherence accuracy.

Effective extraction of 4D dynamic light-field representations.

Enhanced scene flow estimation from sparse light-field data.

Abstract

Light-field video has recently been used in virtual and augmented reality applications to increase realism and immersion. However, existing light-field methods are generally limited to static scenes due to the requirement to acquire a dense scene representation. The large amount of data and the absence of methods to infer temporal coherence pose major challenges in storage, compression and editing compared to conventional video. In this paper, we propose the first method to extract a spatio-temporally coherent light-field video representation. A novel method to obtain Epipolar Plane Images (EPIs) from a spare light-field camera array is proposed. EPIs are used to constrain scene flow estimation to obtain 4D temporally coherent representations of dynamic light-fields. Temporal coherence is achieved on a variety of light-field datasets. Evaluation of the proposed light-field scene flow against existing multi-view dense correspondence approaches demonstrates a significant improvement in accuracy of temporal coherence.