YawDD+: Frame-level Annotations for Accurate Yawn Prediction

TL;DR

This paper introduces YawDD+, a frame-level annotated dataset and models for accurate yawn detection to improve driver fatigue monitoring on edge devices, achieving high accuracy and real-time inference.

Contribution

It presents a semi-automated labeling pipeline for frame-level annotations, enhancing model training and enabling efficient on-device fatigue detection.

Findings

YawDD+ improves frame accuracy by up to 6% over video-level supervision.

Models achieve 99.34% classification accuracy and 95.69% detection mAP.

On-device inference reaches 115 FPS on NVIDIA Jetson AGX.

Abstract

Driver fatigue remains a leading cause of road accidents, responsible for 24% of crashes. While yawning serves as an early behavioral indicator of fatigue, existing approaches face significant challenges due to the presence of systematic noise in video-annotated datasets arising from coarse temporal annotations. Training robust machine learning (ML) models requires rich supervisory labels that help learn salient features from the training data. Moreover, efficient on-device training and inference of models on edge devices is crucial in driver fatigue detection tasks to enable accurate real-time decisions on vehicles without reliance on cloud infrastructure. To address this issue, we develop a semi-automated labeling pipeline with human-in-the-loop verification to annotate YawDD videos to YawDD+ frame-level annotations, enabling more accurate model training on edge platforms such as NVIDIA Jetson NANO. Training the established MNasNet classifier and YOLOv11 detector architectures on YawDD+ improves frame accuracy by up to 6% and mAP by 5% over video-level supervision, achieving 99.34% classification accuracy and 95.69% detection mAP on Jetson NANO and AGX. Moreover, MNasNet completed the epoch time in just 8.69 min/epoch while delivering up to 115 frames-per-second (FPS) inference time on AGX, confirming that enhanced data quality alone supports on-device driver fatigue monitoring systems without server-side computation. The YawDD+ dataset and trained models are available online.