ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop

TL;DR

ModalPatch is a versatile plug-and-play module that enhances the robustness of multi-modal 3D object detection systems against transient modality drops without requiring retraining or architectural modifications.

Contribution

It introduces ModalPatch, a novel, easy-to-integrate module that leverages temporal data and uncertainty-guided fusion to maintain detection performance during modality interruptions.

Findings

Significantly improves detection robustness under modality drop scenarios.

Does not require retraining or architectural changes to existing detectors.

Enhances both robustness and accuracy across multiple detection frameworks.

Abstract

Multi-modal 3D object detection is pivotal for autonomous driving, integrating complementary sensors like LiDAR and cameras. However, its real-world reliability is challenged by transient data interruptions and missing, where modalities can momentarily drop due to hardware glitches, adverse weather, or occlusions. This poses a critical risk, especially during a simultaneous modality drop, where the vehicle is momentarily blind. To address this problem, we introduce ModalPatch, the first plug-and-play module designed to enable robust detection under arbitrary modality-drop scenarios. Without requiring architectural changes or retraining, ModalPatch can be seamlessly integrated into diverse detection frameworks. Technically, ModalPatch leverages the temporal nature of sensor data for perceptual continuity, using a history-based module to predict and compensate for transiently unavailable features. To improve the fidelity of the predicted features, we further introduce an uncertainty-guided cross-modality fusion strategy that dynamically estimates the reliability of compensated features, suppressing biased signals while reinforcing informative ones. Extensive experiments show that ModalPatch consistently enhances both robustness and accuracy of state-of-the-art 3D object detectors under diverse modality-drop conditions.