ThermRad: A Multi-modal Dataset for Robust 3D Object Detection under Challenging Conditions

TL;DR

ThermRad introduces a novel multi-modal dataset with radar, thermal, LiDAR, and camera data captured in extreme weather, alongside a fusion method that significantly improves 3D object detection robustness under challenging conditions.

Contribution

The paper presents the first multi-modal dataset combining radar, thermal, LiDAR, and camera data in extreme weather, and proposes a fusion method that enhances detection accuracy in adverse conditions.

Findings

Detection improvements over 7.98% for cars

Over 24% improvement for pedestrians

Over 27% enhancement for cyclists

Abstract

Robust 3D object detection in extreme weather and illumination conditions is a challenging task. While radars and thermal cameras are known for their resilience to these conditions, few studies have been conducted on radar-thermal fusion due to the lack of corresponding datasets. To address this gap, we first present a new multi-modal dataset called ThermRad, which includes a 3D LiDAR, a 4D radar, an RGB camera and a thermal camera. This dataset is unique because it includes data from all four sensors in extreme weather conditions, providing a valuable resource for future research in this area. To validate the robustness of 4D radars and thermal cameras for 3D object detection in challenging weather conditions, we propose a new multi-modal fusion method called RTDF-RCNN, which leverages the complementary strengths of 4D radars and thermal cameras to boost object detection performance. To further prove the effectiveness of our proposed framework, we re-implement state-of-the-art (SOTA) 3D detectors on our dataset as benchmarks for evaluation. Our method achieves significant enhancements in detecting cars, pedestrians, and cyclists, with improvements of over 7.98%, 24.27%, and 27.15%, respectively, while achieving comparable results to LiDAR-based approaches. Our contributions in both the ThermRad dataset and the new multi-modal fusion method provide a new approach to robust 3D object detection in adverse weather and illumination conditions. The ThermRad dataset will be released.