American Sign Language to Text Translation using Transformer and Seq2Seq with LSTM

TL;DR

This paper compares Transformer and Seq2Seq models for American Sign Language to text translation, showing Transformer outperforms Seq2Seq, but adding ResidualLSTM decreases Transformer performance.

Contribution

It introduces a comparison between Transformer and Seq2Seq models for sign language translation and evaluates the impact of ResidualLSTM on Transformer performance.

Findings

Transformer outperforms Seq2Seq by 28.14 BLEU points.

Adding ResidualLSTM reduces Transformer performance by 23.37%.

Transformer achieves higher translation accuracy than Seq2Seq.

Abstract

Sign language translation is one of the important issues in communication between deaf and hearing people, as it expresses words through hand, body, and mouth movements. American Sign Language is one of the sign languages used, one of which is the alphabetic sign. The development of neural machine translation technology is moving towards sign language translation. Transformer became the state-of-the-art in natural language processing. This study compares the Transformer with the Sequence-to-Sequence (Seq2Seq) model in translating sign language to text. In addition, an experiment was conducted by adding Residual Long Short-Term Memory (ResidualLSTM) in the Transformer. The addition of ResidualLSTM to the Transformer reduces the performance of the Transformer model by 23.37% based on the BLEU Score value. In comparison, the Transformer itself increases the BLEU Score value by 28.14 compared to the Seq2Seq model.