NERVE: A Neuromorphic Vision and Radar Ensemble for Multi-Sensor Fusion Research

TL;DR

NERVE is a comprehensive multi-sensor dataset combining neuromorphic vision and radar data for advancing multi-modal fusion research, with baseline experiments demonstrating improved detection accuracy.

Contribution

The paper introduces NERVE, a large-scale, synchronized neuromorphic vision and radar dataset with annotations, enabling new research in multi-sensor fusion for perception tasks.

Findings

Combining DVS with 77GHz Radar improves human detection accuracy.

Recurrent models achieve up to 47.5% mAP in detection.

Radar distance estimation errors are below 1.8 meters.

Abstract

We present NERVE (Neuromorphic Vision and Radar Ensemble), a multi-sensor dataset comprising 257 minutes of synchronized recordings from five sensors: two Dynamic Vision Sensors (DVS), an RGB-D camera, and two Radar units (24GHz and 77GHz). Captured across 12 measurement days in office environments, NERVE contains around 600GB of uncompressed temporally aligned data with around 914,000 frames and around 9.6 million RGB COCO-formatted annotations covering 16 relevant object categories. To evaluate multi-modal fusion, we construct a DVS+Radar subset for human detection and distance estimation. Baseline experiments using feed-forward and recurrent detectors show that combining DVS with 77GHz Radar consistently improves detection, with recurrent models achieving up to 47.5% mAP and mean absolute Radar distance errors below 1.8m against LiDAR ground truth.