Diffusion-based Light Field Synthesis

TL;DR

LFdiff is a diffusion-based framework that synthesizes light fields from a single RGB image, using disparity estimation and novel conditioning to produce high-quality, controllable light fields for various applications.

Contribution

Introduces LFdiff, a new diffusion model with a position-aware condition scheme and a disentanglement-based noise estimator for light field synthesis from a single image.

Findings

Outperforms existing methods in visual quality and disparity control.

Enhances generalization across different scenes and datasets.

Proves effective in LF super-resolution and refocusing tasks.

Abstract

Light fields (LFs), conducive to comprehensive scene radiance recorded across angular dimensions, find wide applications in 3D reconstruction, virtual reality, and computational photography.However, the LF acquisition is inevitably time-consuming and resource-intensive due to the mainstream acquisition strategy involving manual capture or laborious software synthesis.Given such a challenge, we introduce LFdiff, a straightforward yet effective diffusion-based generative framework tailored for LF synthesis, which adopts only a single RGB image as input.LFdiff leverages disparity estimated by a monocular depth estimation network and incorporates two distinctive components: a novel condition scheme and a noise estimation network tailored for LF data.Specifically, we design a position-aware warping condition scheme, enhancing inter-view geometry learning via a robust conditional signal.We then propose DistgUnet, a disentanglement-based noise estimation network, to harness comprehensive LF representations.Extensive experiments demonstrate that LFdiff excels in synthesizing visually pleasing and disparity-controllable light fields with enhanced generalization capability.Additionally, comprehensive results affirm the broad applicability of the generated LF data, spanning applications like LF super-resolution and refocusing.