Neural Lumigraph Rendering

TL;DR

This paper introduces a neural rendering method that significantly accelerates view synthesis, produces high-quality images, and enables real-time rendering with view-dependent textures by jointly optimizing an implicit surface and radiance field.

Contribution

It presents a novel neural scene representation with periodic activations that speeds up neural volume rendering and allows mesh export with view-dependent textures, bridging neural rendering and traditional graphics.

Findings

Achieves about 100x faster rendering than SOTA methods.

Produces high-quality, photorealistic images from posed 2D images.

Enables real-time rendering with view-dependent textures.

Abstract

Novel view synthesis is a challenging and ill-posed inverse rendering problem. Neural rendering techniques have recently achieved photorealistic image quality for this task. State-of-the-art (SOTA) neural volume rendering approaches, however, are slow to train and require minutes of inference (i.e., rendering) time for high image resolutions. We adopt high-capacity neural scene representations with periodic activations for jointly optimizing an implicit surface and a radiance field of a scene supervised exclusively with posed 2D images. Our neural rendering pipeline accelerates SOTA neural volume rendering by about two orders of magnitude and our implicit surface representation is unique in allowing us to export a mesh with view-dependent texture information. Thus, like other implicit surface representations, ours is compatible with traditional graphics pipelines, enabling real-time rendering rates, while achieving unprecedented image quality compared to other surface methods. We assess the quality of our approach using existing datasets as well as high-quality 3D face data captured with a custom multi-camera rig.