N-DriverMotion: Driver motion learning and prediction using an event-based camera and directly trained spiking neural networks on Loihi 2

TL;DR

This paper introduces N-DriverMotion, a high-resolution event-based driver motion dataset and a novel four-layer spiking neural network for real-time driver gesture recognition, achieving 94.04% accuracy.

Contribution

The paper presents a new high-resolution driver motion dataset and a simplified, directly trainable spiking neural network architecture for efficient, real-time driver gesture recognition.

Findings

Achieved 94.04% accuracy in driver motion recognition.

Developed a high-resolution event-based dataset for driver gestures.

Proposed a four-layer CSNN suitable for on-device real-time inference.

Abstract

Driver motion recognition is a principal factor in ensuring the safety of driving systems. This paper presents a novel system for learning and predicting driver motions and an event-based high-resolution (1280x720) dataset, N-DriverMotion, newly collected to train on a neuromorphic vision system. The system comprises an event-based camera that generates the first high-resolution driver motion dataset representing spike inputs and efficient spiking neural networks (SNNs) that are effective in training and predicting the driver's gestures. The event dataset consists of 13 driver motion categories classified by direction (front, side), illumination (bright, moderate, dark), and participant. A novel simplified four-layer convolutional spiking neural network (CSNN) that we proposed was directly trained using the high-resolution dataset without any time-consuming preprocessing. This enables efficient adaptation to on-device SNNs for real-time inference on high-resolution event-based streams. Compared with recent gesture recognition systems adopting neural networks for vision processing, the proposed neuromorphic vision system achieves comparable accuracy, 94.04\%, in recognizing driver motions with the CSNN architecture. Our proposed CSNN and the dataset can be used to develop safer and more efficient driver monitoring systems for autonomous vehicles or edge devices requiring an efficient neural network architecture.