Lumename: Wearable Device for Hearing Impaired with Personalized ML-Based Auditory Detection and Haptic-Visual Alerts

TL;DR

Lumename is a low-power, real-time smartwatch device that uses personalized machine learning and audio augmentation to detect a user's name and alert them through haptic and visual signals, aiding those with hearing impairments.

Contribution

It introduces a novel audio augmentation technique and an optimized TinyML model for personalized auditory detection on resource-constrained wearable devices.

Findings

Achieved 91.67% accuracy in name detection

Developed a low-resource, low-power TinyML model

Implemented real-time haptic-visual alerts for hearing-impaired users

Abstract

According to the World Health Organization, 430 million people experience disabling hearing loss. For them, recognizing spoken commands such as one's name is difficult. To address this issue, Lumename, a real-time smartwatch, utilizes on-device machine learning to detect a user-customized name before generating a haptic-visual alert. During training, to overcome the need for large datasets, Lumename uses novel audio modulation techniques to augment samples from one user and generate additional samples to represent diverse genders and ages. Constrained random iterations were used to find optimal parameters within the model architecture. This approach resulted in a low-resource and low-power TinyML model that could quickly infer various keyword samples while remaining 91.67\% accurate on a custom-built smartwatch based on an Arduino Nano 33 BLE Sense.