Crowdsampling the Plenoptic Function

TL;DR

This paper introduces DeepMPI, a novel neural rendering method that synthesizes photorealistic views with continuous spatial and lighting variations from unstructured tourist photos, enabling real-time, novel view generation.

Contribution

It presents DeepMPI, a new representation that models time-varying illumination and reflectance from unstructured photo collections for real-time view synthesis.

Findings

Enables real-time photorealistic view synthesis with lighting changes.

Improves over previous neural rendering methods in unstructured data scenarios.

Successfully models reflectance and illumination variations without temporal registration.

Abstract

Many popular tourist landmarks are captured in a multitude of online, public photos. These photos represent a sparse and unstructured sampling of the plenoptic function for a particular scene. In this paper,we present a new approach to novel view synthesis under time-varying illumination from such data. Our approach builds on the recent multi-plane image (MPI) format for representing local light fields under fixed viewing conditions. We introduce a new DeepMPI representation, motivated by observations on the sparsity structure of the plenoptic function, that allows for real-time synthesis of photorealistic views that are continuous in both space and across changes in lighting. Our method can synthesize the same compelling parallax and view-dependent effects as previous MPI methods, while simultaneously interpolating along changes in reflectance and illumination with time. We show how to learn a model of these effects in an unsupervised way from an unstructured collection of photos without temporal registration, demonstrating significant improvements over recent work in neural rendering. More information can be found crowdsampling.io.