DSERT-RoLL: Robust Multi-Modal Perception for Diverse Driving Conditions with Stereo Event-RGB-Thermal Cameras, 4D Radar, and Dual-LiDAR

TL;DR

This paper introduces DSERT-RoLL, a comprehensive multi-modal driving dataset with diverse sensors and conditions, enabling systematic evaluation and fusion strategies for robust perception in autonomous driving.

Contribution

The paper provides a new multi-modal dataset with benchmark protocols and proposes a fusion framework to enhance 3D detection robustness across diverse weather and lighting conditions.

Findings

Established unified 3D and 2D benchmarks for sensor comparison

Reported baseline results for single and multimodal methods

Proposed a sensor fusion framework improving detection robustness

Abstract

In this paper, we present DSERT-RoLL, a driving dataset that incorporates stereo event, RGB, and thermal cameras together with 4D radar and dual LiDAR, collected across diverse weather and illumination conditions. The dataset provides precise 2D and 3D bounding boxes with track IDs and ego vehicle odometry, enabling fair comparisons within and across sensor combinations. It is designed to alleviate data scarcity for novel sensors such as event cameras and 4D radar and to support systematic studies of their behavior. We establish unified 3D and 2D benchmarks that enable direct comparison of characteristics and strengths across sensor families and within each family. We report baselines for representative single modality and multimodal methods and provide protocols that encourage research on different fusion strategies and sensor combinations. In addition, we propose a fusion framework that integrates sensor specific cues into a unified feature space and improves 3D detection robustness under varied weather and lighting.