Pedestrian intention prediction in Adverse Weather Conditions with Spiking Neural Networks and Dynamic Vision Sensors

TL;DR

This paper demonstrates that combining Spiking Neural Networks with Dynamic Vision Sensors significantly enhances pedestrian detection accuracy and efficiency in adverse weather conditions for autonomous vehicles, outperforming traditional CNNs.

Contribution

The study introduces a novel integration of SNNs with DVS for pedestrian detection in challenging weather, showing improved accuracy and computational efficiency over CNNs.

Findings

SNNs with DVS outperform CNNs in accuracy under adverse weather.

SNNs reduce computational overhead compared to traditional CNNs.

SNNs are more efficient for long perception windows and prediction tasks.

Abstract

This study examines the effectiveness of Spiking Neural Networks (SNNs) paired with Dynamic Vision Sensors (DVS) to improve pedestrian detection in adverse weather, a significant challenge for autonomous vehicles. Utilizing the high temporal resolution and low latency of DVS, which excels in dynamic, low-light, and high-contrast environments, we assess the efficiency of SNNs compared to traditional Convolutional Neural Networks (CNNs). Our experiments involved testing across diverse weather scenarios using a custom dataset from the CARLA simulator, mirroring real-world variability. SNN models, enhanced with Temporally Effective Batch Normalization, were trained and benchmarked against state-of-the-art CNNs to demonstrate superior accuracy and computational efficiency in complex conditions such as rain and fog. The results indicate that SNNs, integrated with DVS, significantly reduce computational overhead and improve detection accuracy in challenging conditions compared to CNNs. This highlights the potential of DVS combined with bio-inspired SNN processing to enhance autonomous vehicle perception and decision-making systems, advancing intelligent transportation systems' safety features in varying operational environments. Additionally, our research indicates that SNNs perform more efficiently in handling long perception windows and prediction tasks, rather than simple pedestrian detection.