NeRF for Outdoor Scene Relighting

TL;DR

NeRF-OSR introduces a novel neural radiance field approach for outdoor scene relighting, enabling high-quality, controllable lighting and viewpoint editing from uncontrolled photographs, supported by a new benchmark dataset.

Contribution

It is the first method to achieve outdoor scene relighting with neural radiance fields using only uncontrolled photos, allowing direct illumination control and high-quality editing.

Findings

Outperforms state-of-the-art in relighting quality

Enables realistic self-shadowing effects

Supports simultaneous lighting and viewpoint editing

Abstract

Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination. A delicate manipulation of the lighting can then be performed while keeping the scene albedo and geometry unaltered. We present NeRF-OSR, i.e., the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our technique allows simultaneous editing of both scene illumination and camera viewpoint using only a collection of outdoor photos shot in uncontrolled settings. Moreover, it enables direct control over the scene illumination, as defined through a spherical harmonics model. For evaluation, we collect a new benchmark dataset of several outdoor sites photographed from multiple viewpoints and at different times. For each time, a 360 degree environment map is captured together with a colour-calibration chequerboard to allow accurate numerical evaluations on real data against ground truth. Comparisons against SoTA show that NeRF-OSR enables controllable lighting and viewpoint editing at higher quality and with realistic self-shadowing reproduction. Our method and the dataset are publicly available at https://4dqv.mpi-inf.mpg.de/NeRF-OSR/.