Progressive Radiance Distillation for Inverse Rendering with Gaussian Splatting

TL;DR

This paper introduces progressive radiance distillation, a novel inverse rendering approach that combines physically-based models with Gaussian radiance fields, improving view synthesis and relighting by adaptively balancing physical and learned components.

Contribution

It presents a new progressive distillation framework that leverages a guidance radiance field and a distillation map to enhance inverse rendering accuracy and quality.

Findings

Outperforms state-of-the-art in view synthesis and relighting

Effectively handles unmodeled light paths and artifacts

Applicable to Gaussian splatting and mesh-based methods

Abstract

We propose progressive radiance distillation, an inverse rendering method that combines physically-based rendering with Gaussian-based radiance field rendering using a distillation progress map. Taking multi-view images as input, our method starts from a pre-trained radiance field guidance, and distills physically-based light and material parameters from the radiance field using an image-fitting process. The distillation progress map is initialized to a small value, which favors radiance field rendering. During early iterations when fitted light and material parameters are far from convergence, the radiance field fallback ensures the sanity of image loss gradients and avoids local minima that attracts under-fit states. As fitted parameters converge, the physical model gradually takes over and the distillation progress increases correspondingly. In presence of light paths unmodeled by the physical model, the distillation progress never finishes on affected pixels and the learned radiance field stays in the final rendering. With this designed tolerance for physical model limitations, we prevent unmodeled color components from leaking into light and material parameters, alleviating relighting artifacts. Meanwhile, the remaining radiance field compensates for the limitations of the physical model, guaranteeing high-quality novel views synthesis. Experimental results demonstrate that our method significantly outperforms state-of-the-art techniques quality-wise in both novel view synthesis and relighting. The idea of progressive radiance distillation is not limited to Gaussian splatting. We show that it also has positive effects for prominently specular scenes when adapted to a mesh-based inverse rendering method.