Real-Time Sign Language Gestures to Speech Transcription using Deep Learning

TL;DR

This paper presents a real-time deep learning-based system that translates sign language gestures into speech, improving communication for individuals with hearing and speech impairments through accurate, fast, and user-friendly technology.

Contribution

The study introduces a novel real-time sign language translation system using CNNs trained on Sign Language MNIST, integrating gesture recognition with speech synthesis for practical communication aid.

Findings

High accuracy in gesture classification

Robust real-time performance with low latency

Effective translation into spoken language

Abstract

Communication barriers pose significant challenges for individuals with hearing and speech impairments, often limiting their ability to effectively interact in everyday environments. This project introduces a real-time assistive technology solution that leverages advanced deep learning techniques to translate sign language gestures into textual and audible speech. By employing convolution neural networks (CNN) trained on the Sign Language MNIST dataset, the system accurately classifies hand gestures captured live via webcam. Detected gestures are instantaneously translated into their corresponding meanings and transcribed into spoken language using text-to-speech synthesis, thus facilitating seamless communication. Comprehensive experiments demonstrate high model accuracy and robust real-time performance with some latency, highlighting the system's practical applicability as an accessible, reliable, and user-friendly tool for enhancing the autonomy and integration of sign language users in diverse social settings.