RENI++ A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior

TL;DR

This paper introduces RENI++, a rotation-equivariant, scale-invariant neural illumination prior that accurately models complex HDR environment maps, improving inverse rendering and environment map completion tasks.

Contribution

It presents a novel neural network architecture with built-in equivariance and HDR capabilities, advancing natural illumination modeling for inverse rendering.

Findings

Outperforms traditional illumination representations in expressivity.

Enables accurate environment map completion from partial data.

Demonstrates effectiveness in inverse rendering applications.

Abstract

Inverse rendering is an ill-posed problem. Previous work has sought to resolve this by focussing on priors for object or scene shape or appearance. In this work, we instead focus on a prior for natural illuminations. Current methods rely on spherical harmonic lighting or other generic representations and, at best, a simplistic prior on the parameters. This results in limitations for the inverse setting in terms of the expressivity of the illumination conditions, especially when taking specular reflections into account. We propose a conditional neural field representation based on a variational auto-decoder and a transformer decoder. We extend Vector Neurons to build equivariance directly into our architecture, and leveraging insights from depth estimation through a scale-invariant loss function, we enable the accurate representation of High Dynamic Range (HDR) images. The result is a compact, rotation-equivariant HDR neural illumination model capable of capturing complex, high-frequency features in natural environment maps. Training our model on a curated dataset of 1.6K HDR environment maps of natural scenes, we compare it against traditional representations, demonstrate its applicability for an inverse rendering task and show environment map completion from partial observations. We share our PyTorch implementation, dataset and trained models at https://github.com/JADGardner/ns_reni