NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

TL;DR

NeRV introduces a neural volumetric scene representation that enables relighting and view synthesis under arbitrary lighting by modeling scene properties as continuous functions, improving over prior methods especially in complex lighting conditions.

Contribution

The paper proposes a novel neural scene representation that models scene properties as continuous volumetric functions, allowing for relighting and view synthesis under arbitrary lighting conditions.

Findings

Outperforms existing methods in relighting accuracy.

Handles complex lighting scenarios effectively.

Enables rendering with indirect illumination effects.

Abstract

We present a method that takes as input a set of images of a scene illuminated by unconstrained known lighting, and produces as output a 3D representation that can be rendered from novel viewpoints under arbitrary lighting conditions. Our method represents the scene as a continuous volumetric function parameterized as MLPs whose inputs are a 3D location and whose outputs are the following scene properties at that input location: volume density, surface normal, material parameters, distance to the first surface intersection in any direction, and visibility of the external environment in any direction. Together, these allow us to render novel views of the object under arbitrary lighting, including indirect illumination effects. The predicted visibility and surface intersection fields are critical to our model's ability to simulate direct and indirect illumination during training, because the brute-force techniques used by prior work are intractable for lighting conditions outside of controlled setups with a single light. Our method outperforms alternative approaches for recovering relightable 3D scene representations, and performs well in complex lighting settings that have posed a significant challenge to prior work.