EvDNeRF: Reconstructing Event Data with Dynamic Neural Radiance Fields

TL;DR

EvDNeRF introduces a novel neural radiance field framework that reconstructs and predicts event streams in dynamic scenes, enabling high-fidelity, fast-motion scene modeling from event camera data.

Contribution

It is the first to extend dynamic NeRFs to event data, allowing scene reconstruction and event prediction at fine temporal resolutions.

Findings

Outperforms existing static scene reconstruction methods.

Enables event stream prediction from novel viewpoints.

Provides datasets and code for further research.

Abstract

We present EvDNeRF, a pipeline for generating event data and training an event-based dynamic NeRF, for the purpose of faithfully reconstructing eventstreams on scenes with rigid and non-rigid deformations that may be too fast to capture with a standard camera. Event cameras register asynchronous per-pixel brightness changes at MHz rates with high dynamic range, making them ideal for observing fast motion with almost no motion blur. Neural radiance fields (NeRFs) offer visual-quality geometric-based learnable rendering, but prior work with events has only considered reconstruction of static scenes. Our EvDNeRF can predict eventstreams of dynamic scenes from a static or moving viewpoint between any desired timestamps, thereby allowing it to be used as an event-based simulator for a given scene. We show that by training on varied batch sizes of events, we can improve test-time predictions of events at fine time resolutions, outperforming baselines that pair standard dynamic NeRFs with event generators. We release our simulated and real datasets, as well as code for multi-view event-based data generation and the training and evaluation of EvDNeRF models (https://github.com/anish-bhattacharya/EvDNeRF).