SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weather

TL;DR

SAMFusion introduces a sensor-adaptive multimodal fusion method that effectively combines RGB, LiDAR, NIR, and radar data using attentive, depth-based schemes and transformer decoders to enhance 3D object detection in adverse weather conditions for autonomous vehicles.

Contribution

The paper presents a novel multi-sensor fusion approach that incorporates NIR and radar modalities, with depth-based attention and learned refinement, specifically designed for challenging weather scenarios.

Findings

Improves average precision by 17.2 AP for vulnerable pedestrians at long distances in foggy scenes.

Effectively fuses multiple sensor modalities using attentive, depth-based blending schemes.

Enhances robustness of 3D object detection in adverse weather conditions.

Abstract

Multimodal sensor fusion is an essential capability for autonomous robots, enabling object detection and decision-making in the presence of failing or uncertain inputs. While recent fusion methods excel in normal environmental conditions, these approaches fail in adverse weather, e.g., heavy fog, snow, or obstructions due to soiling. We introduce a novel multi-sensor fusion approach tailored to adverse weather conditions. In addition to fusing RGB and LiDAR sensors, which are employed in recent autonomous driving literature, our sensor fusion stack is also capable of learning from NIR gated camera and radar modalities to tackle low light and inclement weather. We fuse multimodal sensor data through attentive, depth-based blending schemes, with learned refinement on the Bird's Eye View (BEV) plane to combine image and range features effectively. Our detections are predicted by a transformer decoder that weighs modalities based on distance and visibility. We demonstrate that our method improves the reliability of multimodal sensor fusion in autonomous vehicles under challenging weather conditions, bridging the gap between ideal conditions and real-world edge cases. Our approach improves average precision by 17.2 AP compared to the next best method for vulnerable pedestrians in long distances and challenging foggy scenes. Our project page is available at https://light.princeton.edu/samfusion/