RehearsalNeRF: Decoupling Intrinsic Neural Fields of Dynamic Illuminations for Scene Editing

TL;DR

RehearsalNeRF introduces a method to disentangle neural scene representations under dynamic lighting by leveraging stable lighting scenes and optical flow regularization, enabling improved scene editing and view synthesis.

Contribution

It proposes a novel approach using rehearsal scenes and optical flow to effectively disentangle lighting effects in neural radiance fields under dynamic illumination.

Findings

Robust performance in novel view synthesis under changing illumination.

Effective disentanglement of scene radiance and lighting effects.

Ability to reconstruct dynamic scene neural fields with simple masks.

Abstract

Although there has been significant progress in neural radiance fields, an issue on dynamic illumination changes still remains unsolved. Different from relevant works that parameterize time-variant/-invariant components in scenes, subjects' radiance is highly entangled with their own emitted radiance and lighting colors in spatio-temporal domain. In this paper, we present a new effective method to learn disentangled neural fields under the severe illumination changes, named RehearsalNeRF. Our key idea is to leverage scenes captured under stable lighting like rehearsal stages, easily taken before dynamic illumination occurs, to enforce geometric consistency between the different lighting conditions. In particular, RehearsalNeRF employs a learnable vector for lighting effects which represents illumination colors in a temporal dimension and is used to disentangle projected light colors from scene radiance. Furthermore, our RehearsalNeRF is also able to reconstruct the neural fields of dynamic objects by simply adopting off-the-shelf interactive masks. To decouple the dynamic objects, we propose a new regularization leveraging optical flow, which provides coarse supervision for the color disentanglement. We demonstrate the effectiveness of RehearsalNeRF by showing robust performances on novel view synthesis and scene editing under dynamic illumination conditions. Our source code and video datasets will be publicly available.