Learning Sign Language Representation using CNN LSTM, 3DCNN, CNN RNN LSTM and CCN TD

TL;DR

This paper compares various neural network algorithms like CNN, 3DCNN, CNN LSTM, and CCN TD for real-time sign language translation and grading, finding 3DCNN to be the most effective with high accuracy on two datasets.

Contribution

It evaluates and identifies the most effective neural network architecture for real-time sign language translation and grading, using novel datasets.

Findings

3DCNN achieved 91% accuracy on TTSL dataset.

3DCNN achieved 83% accuracy on ASL dataset.

The study compares multiple neural network algorithms for sign language recognition.

Abstract

Existing Sign Language Learning applications focus on the demonstration of the sign in the hope that the student will copy a sign correctly. In these cases, only a teacher can confirm that the sign was completed correctly, by reviewing a video captured manually. Sign Language Translation is a widely explored field in visual recognition. This paper seeks to explore the algorithms that will allow for real-time, video sign translation, and grading of sign language accuracy for new sign language users. This required algorithms capable of recognizing and processing spatial and temporal features. The aim of this paper is to evaluate and identify the best neural network algorithm that can facilitate a sign language tuition system of this nature. Modern popular algorithms including CNN and 3DCNN are compared on a dataset not yet explored, Trinidad and Tobago Sign Language as well as an American Sign Language dataset. The 3DCNN algorithm was found to be the best performing neural network algorithm from these systems with 91% accuracy in the TTSL dataset and 83% accuracy in the ASL dataset.