Neuromorphic Seatbelt State Detection for In-Cabin Monitoring with Event Cameras

TL;DR

This paper demonstrates the feasibility of using neuromorphic event cameras and deep learning for real-time seatbelt state detection in vehicles, achieving high accuracy on synthetic and real data.

Contribution

It introduces a novel application of event-based vision sensors combined with CNNs for seatbelt detection, expanding neuromorphic driver monitoring capabilities.

Findings

Achieved F1 score of 0.989 on synthetic data for seatbelt fastened/unfastened detection.

Achieved F1 score of 0.944 on real data for seatbelt fastened/unfastened detection.

Achieved F1 score of 0.964 on synthetic data for seatbelt fastening/unfastening action classification.

Abstract

Neuromorphic vision sensors, or event cameras, differ from conventional cameras in that they do not capture images at a specified rate. Instead, they asynchronously log local brightness changes at each pixel. As a result, event cameras only record changes in a given scene, and do so with very high temporal resolution, high dynamic range, and low power requirements. Recent research has demonstrated how these characteristics make event cameras extremely practical sensors in driver monitoring systems (DMS), enabling the tracking of high-speed eye motion and blinks. This research provides a proof of concept to expand event-based DMS techniques to include seatbelt state detection. Using an event simulator, a dataset of 108,691 synthetic neuromorphic frames of car occupants was generated from a near-infrared (NIR) dataset, and split into training, validation, and test sets for a seatbelt state detection algorithm based on a recurrent convolutional neural network (CNN). In addition, a smaller set of real event data was collected and reserved for testing. In a binary classification task, the fastened/unfastened frames were identified with an F1 score of 0.989 and 0.944 on the simulated and real test sets respectively. When the problem extended to also classify the action of fastening/unfastening the seatbelt, respective F1 scores of 0.964 and 0.846 were achieved.