Neural Pixel Composition: 3D-4D View Synthesis from Multi-Views

TL;DR

Neural Pixel Composition (NPC) offers a fast, efficient method for continuous 3D-4D view synthesis from sparse multi-view data, outperforming existing approaches in diverse settings and enabling dense reconstruction where others fail.

Contribution

NPC introduces a novel pixel representation and an MLP-based composition method that operates efficiently on sparse multi-view data for high-quality view synthesis.

Findings

Achieves 200-400X faster convergence than existing methods.

Performs better in challenging, sparse multi-view scenarios.

Enables dense 3D reconstruction from limited views.

Abstract

We present Neural Pixel Composition (NPC), a novel approach for continuous 3D-4D view synthesis given only a discrete set of multi-view observations as input. Existing state-of-the-art approaches require dense multi-view supervision and an extensive computational budget. The proposed formulation reliably operates on sparse and wide-baseline multi-view imagery and can be trained efficiently within a few seconds to 10 minutes for hi-res (12MP) content, i.e., 200-400X faster convergence than existing methods. Crucial to our approach are two core novelties: 1) a representation of a pixel that contains color and depth information accumulated from multi-views for a particular location and time along a line of sight, and 2) a multi-layer perceptron (MLP) that enables the composition of this rich information provided for a pixel location to obtain the final color output. We experiment with a large variety of multi-view sequences, compare to existing approaches, and achieve better results in diverse and challenging settings. Finally, our approach enables dense 3D reconstruction from sparse multi-views, where COLMAP, a state-of-the-art 3D reconstruction approach, struggles.