Development of ML model for triboelectric nanogenerator based sign language detection system

TL;DR

This paper compares machine learning and deep learning models for a triboelectric nanogenerator-based sensor glove to recognize sign language gestures, demonstrating that frequency-domain features and multi-sensor architectures significantly improve accuracy.

Contribution

It introduces a multi-sensor MFCC CNN-LSTM architecture for sign language detection using TENG sensors, achieving high accuracy and outperforming traditional ML methods.

Findings

MFCC CNN-LSTM achieves 93.33% accuracy and 95.56% precision.

50-timestep windows balance temporal context and data volume, improving accuracy.

Frequency-domain features and parallel sensor processing enhance gesture recognition performance.

Abstract

Sign language recognition (SLR) is vital for bridging communication gaps between deaf and hearing communities. Vision-based approaches suffer from occlusion, computational costs, and physical constraints. This work presents a comparison of machine learning (ML) and deep learning models for a custom triboelectric nanogenerator (TENG)-based sensor glove. Utilizing multivariate time-series data from five flex sensors, the study benchmarks traditional ML algorithms, feedforward neural networks, LSTM-based temporal models, and a multi-sensor MFCC CNN-LSTM architecture across 11 sign classes (digits 1-5, letters A-F). The proposed MFCC CNN-LSTM architecture processes frequency-domain features from each sensor through independent convolutional branches before fusion. It achieves 93.33% accuracy and 95.56% precision, a 23-point improvement over the best ML algorithm (Random Forest: 70.38%). Ablation studies reveal 50-timestep windows offer a tradeoff between temporal context and training data volume, yielding 84.13% accuracy compared to 58.06% with 100-timestep windows. MFCC feature extraction maps temporal variations to execution-speed-invariant spectral representations, and data augmentation methods (time warping, noise injection) are essential for generalization. Results demonstrate that frequency-domain feature representations combined with parallel multi-sensor processing architectures offer enhancement over classical algorithms and time-domain deep learning for wearable sensor-based gesture recognition. This aids assistive technology development.