Robust e-NeRF: NeRF from Sparse & Noisy Events under Non-Uniform Motion

TL;DR

Robust e-NeRF introduces a method to reconstruct Neural Radiance Fields from sparse, noisy event camera data under non-uniform motion, overcoming previous limitations related to noise, sparsity, and varying camera parameters.

Contribution

The paper presents a novel approach with a realistic event generation model and generalized reconstruction losses, enabling robust NeRF reconstruction from challenging event data.

Findings

Effective reconstruction from sparse, noisy events demonstrated

Generalizes across different motion speeds and intrinsic parameters

Validated on real and simulated sequences

Abstract

Event cameras offer many advantages over standard cameras due to their distinctive principle of operation: low power, low latency, high temporal resolution and high dynamic range. Nonetheless, the success of many downstream visual applications also hinges on an efficient and effective scene representation, where Neural Radiance Field (NeRF) is seen as the leading candidate. Such promise and potential of event cameras and NeRF inspired recent works to investigate on the reconstruction of NeRF from moving event cameras. However, these works are mainly limited in terms of the dependence on dense and low-noise event streams, as well as generalization to arbitrary contrast threshold values and camera speed profiles. In this work, we propose Robust e-NeRF, a novel method to directly and robustly reconstruct NeRFs from moving event cameras under various real-world conditions, especially from sparse and noisy events generated under non-uniform motion. It consists of two key components: a realistic event generation model that accounts for various intrinsic parameters (e.g. time-independent, asymmetric threshold and refractory period) and non-idealities (e.g. pixel-to-pixel threshold variation), as well as a complementary pair of normalized reconstruction losses that can effectively generalize to arbitrary speed profiles and intrinsic parameter values without such prior knowledge. Experiments on real and novel realistically simulated sequences verify our effectiveness. Our code, synthetic dataset and improved event simulator are public.