Speech Tasks Relevant to Sleepiness Determined with Deep Transfer Learning

TL;DR

This study develops a deep transfer learning model using HuBERT speech representations to detect sleepiness from speech data, identifying specific speech tasks that improve detection accuracy, which could aid in preventing sleep-related accidents.

Contribution

The paper introduces a novel approach using deep transfer learning on speech tasks to detect sleepiness, highlighting the importance of specific tasks like memory recall and naming.

Findings

Best model achieved 80-81% accuracy.

Memory recall and naming tasks are most informative.

Speech-based sleepiness detection is promising and cost-effective.

Abstract

Excessive sleepiness in attention-critical contexts can lead to adverse events, such as car crashes. Detecting and monitoring sleepiness can help prevent these adverse events from happening. In this paper, we use the Voiceome dataset to extract speech from 1,828 participants to develop a deep transfer learning model using Hidden-Unit BERT (HuBERT) speech representations to detect sleepiness from individuals. Speech is an under-utilized source of data in sleep detection, but as speech collection is easy, cost-effective, and non-invasive, it provides a promising resource for sleepiness detection. Two complementary techniques were conducted in order to seek converging evidence regarding the importance of individual speech tasks. Our first technique, masking, evaluated task importance by combining all speech tasks, masking selected responses in the speech, and observing systematic changes in model accuracy. Our second technique, separate training, compared the accuracy of multiple models, each of which used the same architecture, but was trained on a different subset of speech tasks. Our evaluation shows that the best-performing model utilizes the memory recall task and categorical naming task from the Boston Naming Test, which achieved an accuracy of 80.07% (F1-score of 0.85) and 81.13% (F1-score of 0.89), respectively.