ASL Trigger Recognition in Mixed Activity/Signing Sequences for RF Sensor-Based User Interfaces

TL;DR

This paper presents a novel RF sensor-based method for recognizing sign language trigger signs within mixed activity sequences, achieving high accuracy and addressing privacy and lighting limitations of video-based recognition.

Contribution

It introduces a multi-representation RF data approach for sequential classification of sign language triggers amidst daily activities, improving recognition in real-world settings.

Findings

Achieved 98.9% trigger sign detection rate.

Attained 92% classification accuracy for 15 ASL words.

Effectively distinguished signs from gross motor activities.

Abstract

The past decade has seen great advancements in speech recognition for control of interactive devices, personal assistants, and computer interfaces. However, Deaf and hard-ofhearing (HoH) individuals, whose primary mode of communication is sign language, cannot use voice-controlled interfaces. Although there has been significant work in video-based sign language recognition, video is not effective in the dark and has raised privacy concerns in the Deaf community when used in the context of human ambient intelligence. RF sensors have been recently proposed as a new modality that can be effective under the circumstances where video is not. This paper considers the problem of recognizing a trigger sign (wake word) in the context of daily living, where gross motor activities are interwoven with signing sequences. The proposed approach exploits multiple RF data domain representations (time-frequency, range-Doppler, and range-angle) for sequential classification of mixed motion data streams. The recognition accuracy of signs with varying kinematic properties is compared and used to make recommendations on appropriate trigger sign selection for RFsensor based user interfaces. The proposed approach achieves a trigger sign detection rate of 98.9% and a classification accuracy of 92% for 15 ASL words and 3 gross motor activities.