Sign Language Recognition using Parallel Bidirectional Reservoir Computing

TL;DR

This paper introduces a lightweight sign language recognition system combining parallel bidirectional reservoir computing with MediaPipe for real-time, edge-device deployment, achieving high accuracy with minimal training time.

Contribution

It presents a novel PBRC architecture that captures temporal dependencies efficiently, reducing training time significantly compared to traditional deep learning methods.

Findings

Achieved top-1 accuracy of 60.85% on WLASL dataset.

Training time reduced to 18.67 seconds.

Outperformed deep learning methods in training efficiency.

Abstract

Sign language recognition (SLR) facilitates communication between deaf and hearing communities. Deep learning based SLR models are commonly used but require extensive computational resources, making them unsuitable for deployment on edge devices. To address these limitations, we propose a lightweight SLR system that combines parallel bidirectional reservoir computing (PBRC) with MediaPipe. MediaPipe enables real-time hand tracking and precise extraction of hand joint coordinates, which serve as input features for the PBRC architecture. The proposed PBRC architecture consists of two echo state network (ESN) based bidirectional reservoir computing (BRC) modules arranged in parallel to capture temporal dependencies, thereby creating a rich feature representation for classification. We trained our PBRC-based SLR system on the Word-Level American Sign Language (WLASL) video dataset, achieving top-1, top-5, and top-10 accuracies of 60.85%, 85.86%, and 91.74%, respectively. Training time was significantly reduced to 18.67 seconds due to the intrinsic properties of reservoir computing, compared to over 55 minutes for deep learning based methods such as Bi-GRU. This approach offers a lightweight, cost-effective solution for real-time SLR on edge devices.