RealLiFe: Real-Time Light Field Reconstruction via Hierarchical Sparse Gradient Descent

TL;DR

RealLiFe is a real-time light field reconstruction method that uses hierarchical sparse gradient descent to produce high-quality results from sparse inputs, significantly outperforming existing methods in speed and quality.

Contribution

The paper introduces a novel hierarchical sparse gradient descent technique enabling real-time light field reconstruction from sparse views, with high quality and efficiency.

Findings

Achieves 100x faster reconstruction than offline methods.

Produces higher PSNR (about 2 dB) than other online approaches.

Maintains comparable visual quality with significantly reduced computation time.

Abstract

With the rise of Extended Reality (XR) technology, there is a growing need for real-time light field reconstruction from sparse view inputs. Existing methods can be classified into offline techniques, which can generate high-quality novel views but at the cost of long inference/training time, and online methods, which either lack generalizability or produce unsatisfactory results. However, we have observed that the intrinsic sparse manifold of Multi-plane Images (MPI) enables a significant acceleration of light field reconstruction while maintaining rendering quality.Based on this insight, we introduce \textbf{RealLiFe}, a novel light field optimization method, which leverages the proposed Hierarchical Sparse Gradient Descent (HSGD) to produce high-quality light fields from sparse input images in real time. Technically, the coarse MPI of a scene is first generated using a 3D CNN, and it is further optimized leveraging only the scene content aligned sparse MPI gradients in a few iterations. Extensive experiments demonstrate that our method achieves comparable visual quality while being 100x faster on average than state-of-the-art offline methods and delivers better performance (about 2 dB higher in PSNR) compared to other online approaches.