Dynamic EventNeRF: Reconstructing General Dynamic Scenes from Multi-view RGB and Event Streams

TL;DR

This paper introduces Dynamic EventNeRF, a novel method for reconstructing dynamic scenes using multi-view RGB and event streams, excelling in low-light and fast-motion conditions by leveraging event cameras and NeRF models.

Contribution

It is the first to combine multi-view event streams with sparse RGB frames for scene reconstruction, advancing beyond traditional RGB-only methods.

Findings

Outperforms RGB-based baselines in challenging conditions

Produces state-of-the-art multi-view dynamic scene reconstructions

Introduces a new dataset with multi-view event streams and challenging motions

Abstract

Volumetric reconstruction of dynamic scenes is an important problem in computer vision. It is especially challenging in poor lighting and with fast motion. This is partly due to limitations of RGB cameras: To capture frames under low lighting, the exposure time needs to be increased, which leads to more motion blur. In contrast, event cameras, which record changes in pixel brightness asynchronously, are much less dependent on lighting, making them more suitable for recording fast motion. We hence propose the first method to spatiotemporally reconstruct a scene from sparse multi-view event streams and sparse RGB frames. We train a sequence of cross-faded time-conditioned NeRF models, one per short recording segment. The individual segments are supervised with a set of event- and RGB-based losses and sparse-view regularisation. We assemble a real-world multi-view camera rig with six static event cameras around the object and record a benchmark multi-view event stream dataset of challenging motions. Our work outperforms RGB-based baselines, producing state-of-the-art results, and opens up the topic of multi-view event-based reconstruction as a new path for fast scene capture beyond RGB cameras. The code and the data are released at https://4dqv.mpi-inf.mpg.de/DynEventNeRF/