Neural Free-Viewpoint Relighting for Glossy Indirect Illumination

TL;DR

This paper introduces a neural-wavelet hybrid approach for precomputed radiance transfer that enables real-time relighting of scenes with high-frequency, glossy, and view-dependent indirect illumination effects, surpassing previous low-frequency limitations.

Contribution

It presents a novel neural-wavelet PRT method that captures high-frequency global illumination, including glossy reflections, for dynamic relighting with changing viewpoints.

Findings

Achieves real-time rendering at 24 FPS for 512x512 resolution.

Handles complex effects like caustics and view-dependent reflections.

Outperforms traditional low-frequency PRT methods in high-frequency scenarios.

Abstract

Precomputed Radiance Transfer (PRT) remains an attractive solution for real-time rendering of complex light transport effects such as glossy global illumination. After precomputation, we can relight the scene with new environment maps while changing viewpoint in real-time. However, practical PRT methods are usually limited to low-frequency spherical harmonic lighting. All-frequency techniques using wavelets are promising but have so far had little practical impact. The curse of dimensionality and much higher data requirements have typically limited them to relighting with fixed view or only direct lighting with triple product integrals. In this paper, we demonstrate a hybrid neural-wavelet PRT solution to high-frequency indirect illumination, including glossy reflection, for relighting with changing view. Specifically, we seek to represent the light transport function in the Haar wavelet basis. For global illumination, we learn the wavelet transport using a small multi-layer perceptron (MLP) applied to a feature field as a function of spatial location and wavelet index, with reflected direction and material parameters being other MLP inputs. We optimize/learn the feature field (compactly represented by a tensor decomposition) and MLP parameters from multiple images of the scene under different lighting and viewing conditions. We demonstrate real-time (512 x 512 at 24 FPS, 800 x 600 at 13 FPS) precomputed rendering of challenging scenes involving view-dependent reflections and even caustics.