Light Field Diffusion for Single-View Novel View Synthesis

TL;DR

Light Field Diffusion (LFD) introduces a novel diffusion-based method for single-view novel view synthesis that enhances 3D consistency by using light field encoding instead of camera pose matrices, achieving high-fidelity and generalizable results.

Contribution

LFD transforms camera pose matrices into light field encoding to impose local constraints, improving view consistency in diffusion-based NVS without relying solely on pose matrices.

Findings

LFD outperforms existing methods in view consistency and image quality.

LFD demonstrates strong zero-shot generalization to out-of-distribution datasets.

LFD achieves high-fidelity synthesis in complex textured regions.

Abstract

Single-view novel view synthesis (NVS), the task of generating images from new viewpoints based on a single reference image, is important but challenging in computer vision. Recent advancements in NVS have leveraged Denoising Diffusion Probabilistic Models (DDPMs) for their exceptional ability to produce high-fidelity images. However, current diffusion-based methods typically utilize camera pose matrices to globally and implicitly enforce 3D constraints, which can lead to inconsistencies in images generated from varying viewpoints, particularly in regions with complex textures and structures. To address these limitations, we present Light Field Diffusion (LFD), a novel conditional diffusion-based approach that transcends the conventional reliance on camera pose matrices. Starting from the camera pose matrices, LFD transforms them into light field encoding, with the same shape as the reference image, to describe the direction of each ray. By integrating light field encoding with the reference image, our method imposes local pixel-wise constraints within the diffusion process, fostering enhanced view consistency. Our approach not only involves training image LFD on the ShapeNet Car dataset but also includes fine-tuning a pre-trained latent diffusion model on the Objaverse dataset. This enables our latent LFD model to exhibit remarkable zero-shot generalization capabilities across out-of-distribution datasets like RTMV as well as in-the-wild images. Experiments demonstrate that LFD not only produces high-fidelity images but also achieves superior 3D consistency in complex regions, outperforming existing novel view synthesis methods.