Neural Ray-Tracing: Learning Surfaces and Reflectance for Relighting and View Synthesis

TL;DR

This paper introduces a neural rendering approach that explicitly models light transport between surfaces, enabling scene editing, relighting, and reflectance estimation by separating scene components unlike previous volumetric methods.

Contribution

It presents a method that explicitly models scene light transport using traditional rendering equations, allowing for scene editing and relighting with learned surface and reflectance properties.

Findings

Outperforms NeRV in relighting accuracy under known lighting.

Produces realistic scene reconstructions for relit and edited scenes.

Enables BSDF recovery and scene editing from synthetic and real data.

Abstract

Recent neural rendering methods have demonstrated accurate view interpolation by predicting volumetric density and color with a neural network. Although such volumetric representations can be supervised on static and dynamic scenes, existing methods implicitly bake the complete scene light transport into a single neural network for a given scene, including surface modeling, bidirectional scattering distribution functions, and indirect lighting effects. In contrast to traditional rendering pipelines, this prohibits changing surface reflectance, illumination, or composing other objects in the scene. In this work, we explicitly model the light transport between scene surfaces and we rely on traditional integration schemes and the rendering equation to reconstruct a scene. The proposed method allows BSDF recovery with unknown light conditions and classic light transports such as pathtracing. By learning decomposed transport with surface representations established in conventional rendering methods, the method naturally facilitates editing shape, reflectance, lighting and scene composition. The method outperforms NeRV for relighting under known lighting conditions, and produces realistic reconstructions for relit and edited scenes. We validate the proposed approach for scene editing, relighting and reflectance estimation learned from synthetic and captured views on a subset of NeRV's datasets.