# A framework for detecting and predicting highway traffic anomalies via multimodal fusion and heterogeneous graph neural networks

## TL;DR

This paper introduces a new method for detecting and predicting highway traffic anomalies using multimodal data and graph neural networks, improving accuracy and stability over existing models.

## Contribution

The novel framework combines multimodal fusion and heterogeneous graph neural networks with an Ensemble CPLE algorithm for enhanced traffic anomaly detection.

## Key findings

- The MHGNN-CPLE model outperforms existing models like AGC-LSTM and AttentionDeepST in accuracy and F1 score.
- The model maintains high accuracy under varying noise levels in dynamic detection scenarios.
- Multimodal data integration and HGNNs effectively capture complex spatiotemporal dependencies in traffic events.

## Abstract

This paper presents a novel framework for detecting and predicting abnormal traffic events on highways. Current traffic monitoring systems often rely on single data sources, which limits their detection accuracy and robustness in complex environments. To address these limitations, we propose a framework based on multimodal deep fusion and heterogeneous graph neural networks (HGNNs), incorporating an Ensemble Contrastive Pessimistic Likelihood Estimation (CPLE) algorithm to optimize performance. The framework integrates static and dynamic traffic data, such as video images, traffic flow, vehicle speed, and tunnel weather conditions. Experimental results demonstrate that the model performs well in various scenarios, showing significant improvement in accuracy and stability over existing models like AGC-LSTM and AttentionDeepST. For instance, the proposed MHGNN-CPLE model achieves superior accuracy and F1 score in static detection tasks while maintaining high accuracy under different noise levels in dynamic detection scenarios. This study provides a significant advancement in traffic event analysis by effectively combining multimodal data and leveraging HGNNs to capture complex spatiotemporal dependencies.