Driver Assistance System Based on Multimodal Data Hazard Detection

TL;DR

This paper introduces a multimodal driver assistance system that combines video and audio data with an attention-based fusion strategy, improving the detection of driving anomalies and enhancing safety in autonomous driving scenarios.

Contribution

It presents a novel multimodal fusion approach and a new dataset for better detection of rare driving incidents in autonomous vehicles.

Findings

Improved incident recognition accuracy with multimodal data

Effective cross-modal correlation capture reduces misjudgments

New three-modality dataset enhances research in driving anomaly detection

Abstract

Autonomous driving technology has advanced significantly, yet detecting driving anomalies remains a major challenge due to the long-tailed distribution of driving events. Existing methods primarily rely on single-modal road condition video data, which limits their ability to capture rare and unpredictable driving incidents. This paper proposes a multimodal driver assistance detection system that integrates road condition video, driver facial video, and audio data to enhance incident recognition accuracy. Our model employs an attention-based intermediate fusion strategy, enabling end-to-end learning without separate feature extraction. To support this approach, we develop a new three-modality dataset using a driving simulator. Experimental results demonstrate that our method effectively captures cross-modal correlations, reducing misjudgments and improving driving safety.