Effect of Kinematics and Fluency in Adversarial Synthetic Data Generation for ASL Recognition with RF Sensors

TL;DR

This paper investigates how signing kinematics and fluency affect RF sensor-based ASL recognition, proposing two synthetic data generation methods that improve deep learning performance in sign language classification.

Contribution

It introduces two novel approaches for synthetic RF sign language data generation, addressing the impact of signing fluency and kinematics on recognition accuracy.

Findings

Kinematic differences between fluent and imitation signers are significant.

Training on fluent signer data yields higher recognition accuracy than adaptation methods.

Synthetic data from fluent signers achieves 93% top-5 accuracy in ASL classification.

Abstract

RF sensors have been recently proposed as a new modality for sign language processing technology. They are non-contact, effective in the dark, and acquire a direct measurement of signing kinematic via exploitation of the micro-Doppler effect. First, this work provides an in depth, comparative examination of the kinematic properties of signing as measured by RF sensors for both fluent ASL users and hearing imitation signers. Second, as ASL recognition techniques utilizing deep learning requires a large amount of training data, this work examines the effect of signing kinematics and subject fluency on adversarial learning techniques for data synthesis. Two different approaches for the synthetic training data generation are proposed: 1) adversarial domain adaptation to minimize the differences between imitation signing and fluent signing data, and 2) kinematically-constrained generative adversarial networks for accurate synthesis of RF signing signatures. The results show that the kinematic discrepancies between imitation signing and fluent signing are so significant that training on data directly synthesized from fluent RF signers offers greater performance (93% top-5 accuracy) than that produced by adaptation of imitation signing (88% top-5 accuracy) when classifying 100 ASL signs.