Ev-NeRF: Event Based Neural Radiance Field

TL;DR

Ev-NeRF introduces a neural radiance field model that effectively reconstructs images from noisy event camera data by leveraging multi-view consistency, enabling high-quality imaging in challenging conditions.

Contribution

This work is the first to adapt Neural Radiance Fields to event camera data, using multi-view consistency for self-supervision to handle noise and sparse measurements.

Findings

Achieves competitive intensity image reconstruction under extreme noise.

Provides high-quality novel view synthesis from sparse event data.

Enables depth estimation from event-based neural volumes.

Abstract

We present Ev-NeRF, a Neural Radiance Field derived from event data. While event cameras can measure subtle brightness changes in high frame rates, the measurements in low lighting or extreme motion suffer from significant domain discrepancy with complex noise. As a result, the performance of event-based vision tasks does not transfer to challenging environments, where the event cameras are expected to thrive over normal cameras. We find that the multi-view consistency of NeRF provides a powerful self-supervision signal for eliminating the spurious measurements and extracting the consistent underlying structure despite highly noisy input. Instead of posed images of the original NeRF, the input to Ev-NeRF is the event measurements accompanied by the movements of the sensors. Using the loss function that reflects the measurement model of the sensor, Ev-NeRF creates an integrated neural volume that summarizes the unstructured and sparse data points captured for about 2-4 seconds. The generated neural volume can also produce intensity images from novel views with reasonable depth estimates, which can serve as a high-quality input to various vision-based tasks. Our results show that Ev-NeRF achieves competitive performance for intensity image reconstruction under extreme noise conditions and high-dynamic-range imaging.