Sign Language Recognition using Bidirectional Reservoir Computing

TL;DR

This paper introduces an efficient sign language recognition system that uses MediaPipe and bidirectional reservoir computing, achieving competitive accuracy with minimal training time, suitable for resource-limited devices.

Contribution

It presents a novel BRC-based SLR approach combining MediaPipe and ESN, significantly reducing training time compared to deep learning methods.

Findings

Achieved 57.71% accuracy on WLASL dataset.

Training time reduced to 9 seconds from 55 minutes.

Suitable for deployment on resource-constrained devices.

Abstract

Sign language recognition (SLR) facilitates communication between deaf and hearing individuals. Deep learning is widely used to develop SLR-based systems; however, it is computationally intensive and requires substantial computational resources, making it unsuitable for resource-constrained devices. To address this, we propose an efficient sign language recognition system using MediaPipe and an echo state network (ESN)-based bidirectional reservoir computing (BRC) architecture. MediaPipe extracts hand joint coordinates, which serve as inputs to the ESN-based BRC architecture. The BRC processes these features in both forward and backward directions, efficiently capturing temporal dependencies. The resulting states of BRC are concatenated to form a robust representation for classification. We evaluated our method on the Word-Level American Sign Language (WLASL) video dataset, achieving a competitive accuracy of 57.71% and a significantly lower training time of only 9 seconds, in contrast to the 55 minutes and $38$ seconds required by the deep learning-based Bi-GRU approach. Consequently, the BRC-based SLR system is well-suited for edge devices.