Joint Optimization of Triangle Mesh, Material, and Light from Neural Fields with Neural Radiance Cache

TL;DR

This paper presents an efficient inverse rendering framework that combines neural fields and a neural radiance cache to accurately reconstruct triangle meshes, materials, and lighting while modeling global illumination through differentiable ray tracing.

Contribution

It introduces a novel method that integrates neural fields with a neural radiance cache for joint optimization of geometry, material, and lighting.

Findings

Effective prevention of indirect illumination baking into materials

High-quality reconstruction of triangle meshes, PBR materials, and HDR light probes

Utilization of neural radiance cache enhances global illumination modeling

Abstract

Traditional inverse rendering techniques are based on textured meshes, which naturally adapts to modern graphics pipelines, but costly differentiable multi-bounce Monte Carlo (MC) ray tracing poses challenges for modeling global illumination. Recently, neural fields has demonstrated impressive reconstruction quality but falls short in modeling indirect illumination. In this paper, we introduce a simple yet efficient inverse rendering framework that combines the strengths of both methods. Specifically, given pre-trained neural field representing the scene, we can obtain an initial estimate of the signed distance field (SDF) and create a Neural Radiance Cache (NRC), an enhancement over the traditional radiance cache used in real-time rendering. By using the former to initialize differentiable marching tetrahedrons (DMTet) and the latter to model indirect illumination, we can compute the global illumination via single-bounce differentiable MC ray tracing and jointly optimize the geometry, material, and light through back propagation. Experiments demonstrate that, compared to previous methods, our approach effectively prevents indirect illumination effects from being baked into materials, thus obtaining the high-quality reconstruction of triangle mesh, Physically-Based (PBR) materials, and High Dynamic Range (HDR) light probe.