Real-time Rendering with a Neural Irradiance Volume

TL;DR

The paper introduces Neural Irradiance Volume (NIV), a neural-based method for real-time diffuse global illumination rendering that significantly reduces memory usage and artifacts compared to traditional probe-based approaches.

Contribution

NIV provides a neural compression technique that enables accurate, memory-efficient, and real-time rendering of dynamic global illumination without complex heuristics or ray tracing.

Findings

Memory usage reduced by at least 10x at the same quality

Achieves around 1 ms per frame inference on consumer GPUs

Supports rendering of time-varying effects without extra computation

Abstract

Rendering diffuse global illumination in real-time is often approximated by pre-computing and storing irradiance in a 3D grid of probes. As long as most of the scene remains static, probes approximate irradiance for all surfaces immersed in the irradiance volume, including novel dynamic objects. This approach, however, suffers from aliasing artifacts and high memory consumption. We propose Neural Irradiance Volume (NIV), a neural-based technique that allows accurate real-time rendering of diffuse global illumination via a compact pre-computed model, overcoming the limitations of traditional probe-based methods, such as the expensive memory footprint, aliasing artifacts, and scene-specific heuristics. The key insight is that neural compression creates an adaptive and amortized representation of irradiance, circumventing the cubic scaling of grid-based methods. Our superior memory-scaling improves quality by at least 10x at the same memory budget, and enables a straightforward representation of higher-dimensional irradiance fields, allowing rendering of time-varying or dynamic effects without requiring additional computation at runtime. Unlike other neural rendering techniques, our method works within strict real-time constraints, providing fast inference (around 1 ms per frame on consumer GPUs at full HD resolution), reduced memory usage (1-5 MB for medium-sized scenes), and only requires a G-buffer as input, without expensive ray tracing or denoising.