Multi-Modal Sensor Fusion for Proactive Blockage Prediction in mmWave Vehicular Networks

TL;DR

This paper presents a multi-modal sensor fusion framework using camera, GPS, LiDAR, and radar to proactively predict signal blockage in mmWave vehicular networks, enhancing communication reliability.

Contribution

It introduces a novel multi-modal deep learning approach with modality-specific models and a softmax ensemble for proactive blockage prediction in vehicular environments.

Findings

Camera-only model achieves 97.1% F1-score with 89.8ms inference time.

Combining camera and radar improves accuracy to 97.2% F1 at 95.7ms.

Multi-modal sensing effectively predicts blockages up to 1.5 seconds in advance.

Abstract

Vehicular communication systems operating in the millimeter wave (mmWave) band are highly susceptible to signal blockage from dynamic obstacles such as vehicles, pedestrians, and infrastructure. To address this challenge, we propose a proactive blockage prediction framework that utilizes multi-modal sensing, including camera, GPS, LiDAR, and radar inputs in an infrastructure-to-vehicle (I2V) setting. This approach uses modality-specific deep learning models to process each sensor stream independently and fuses their outputs using a softmax-weighted ensemble strategy based on validation performance. Our evaluations, for up to 1.5s in advance, show that the camera-only model achieves the best standalone trade-off with an F1-score of 97.1% and an inference time of 89.8ms. A camera+radar configuration further improves accuracy to 97.2% F1 at 95.7ms. Our results display the effectiveness and efficiency of multi-modal sensing for mmWave blockage prediction and provide a pathway for proactive wireless communication in dynamic environments.