Geo-NI: Geometry-aware Neural Interpolation for Light Field Rendering

TL;DR

Geo-NI introduces a novel light field rendering framework that combines neural interpolation with a depth hypothesis-based DIBR pipeline, enabling high-quality view synthesis with large disparities and non-Lambertian effects.

Contribution

It integrates neural interpolation with a depth-based DIBR approach using a reconstruction cost volume, enhancing light field rendering accuracy and robustness.

Findings

Outperforms existing methods on various datasets.

Effectively handles large disparities and non-Lambertian effects.

Demonstrates superior rendering quality in experiments.

Abstract

In this paper, we present a Geometry-aware Neural Interpolation (Geo-NI) framework for light field rendering. Previous learning-based approaches either rely on the capability of neural networks to perform direct interpolation, which we dubbed Neural Interpolation (NI), or explore scene geometry for novel view synthesis, also known as Depth Image-Based Rendering (DIBR). Instead, we incorporate the ideas behind these two kinds of approaches by launching the NI with a novel DIBR pipeline. Specifically, the proposed Geo-NI first performs NI using input light field sheared by a set of depth hypotheses. Then the DIBR is implemented by assigning the sheared light fields with a novel reconstruction cost volume according to the reconstruction quality under different depth hypotheses. The reconstruction cost is interpreted as a blending weight to render the final output light field by blending the reconstructed light fields along the dimension of depth hypothesis. By combining the superiorities of NI and DIBR, the proposed Geo-NI is able to render views with large disparity with the help of scene geometry while also reconstruct non-Lambertian effect when depth is prone to be ambiguous. Extensive experiments on various datasets demonstrate the superior performance of the proposed geometry-aware light field rendering framework.