Diffusion-Based Restoration for Multi-Modal 3D Object Detection in Adverse Weather

TL;DR

DiffFusion is a diffusion-based framework that enhances multi-modal 3D object detection robustness in adverse weather by restoring degraded data and aligning modalities, outperforming existing methods in challenging conditions.

Contribution

The paper introduces DiffFusion, a novel diffusion-based approach with adaptive fusion and alignment modules for robust multi-modal 3D detection in adverse weather.

Findings

Achieves state-of-the-art robustness under adverse weather conditions.

Maintains high performance on clean data.

Demonstrates strong generalization in zero-shot real-world tests.

Abstract

Multi-modal 3D object detection is important for reliable perception in robotics and autonomous driving. However, its effectiveness remains limited under adverse weather conditions due to weather-induced distortions and misalignment between different data modalities. In this work, we propose DiffFusion, a novel framework designed to enhance robustness in challenging weather through diffusion-based restoration and adaptive cross-modal fusion. Our key insight is that diffusion models possess strong capabilities for denoising and generating data that can adapt to various weather conditions. Building on this, DiffFusion introduces Diffusion-IR restoring images degraded by weather effects and Point Cloud Restoration (PCR) compensating for corrupted LiDAR data using image object cues. To tackle misalignments between two modalities, we develop Bidirectional Adaptive Fusion and Alignment Module (BAFAM). It enables dynamic multi-modal fusion and bidirectional bird's-eye view (BEV) alignment to maintain consistent spatial correspondence. Extensive experiments on three public datasets show that DiffFusion achieves state-of-the-art robustness under adverse weather while preserving strong clean-data performance. Zero-shot results on the real-world DENSE dataset further validate its generalization. The implementation of our DiffFusion will be released as open-source.