When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network

TL;DR

This paper presents a real-time anomaly detection system for autonomous driving that combines event camera data with RGB images using a novel asynchronous hybrid network, achieving high accuracy and millisecond response times.

Contribution

It introduces a multimodal asynchronous hybrid network that effectively fuses event streams and RGB images for fast, accurate anomaly detection in autonomous driving.

Findings

Outperforms existing methods in accuracy and response time

Achieves millisecond-level real-time performance

Effectively captures temporal dynamics and spatial details

Abstract

Anomaly detection is essential for the safety and reliability of autonomous driving systems. Current methods often focus on detection accuracy but neglect response time, which is critical in time-sensitive driving scenarios. In this paper, we introduce real-time anomaly detection for autonomous driving, prioritizing both minimal response time and high accuracy. We propose a novel multimodal asynchronous hybrid network that combines event streams from event cameras with image data from RGB cameras. Our network utilizes the high temporal resolution of event cameras through an asynchronous Graph Neural Network and integrates it with spatial features extracted by a CNN from RGB images. This combination effectively captures both the temporal dynamics and spatial details of the driving environment, enabling swift and precise anomaly detection. Extensive experiments on benchmark datasets show that our approach outperforms existing methods in both accuracy and response time, achieving millisecond-level real-time performance.