Spatio-temporal transformer to support automatic sign language translation

TL;DR

This paper presents a spatio-temporal transformer architecture for sign language translation that effectively captures gesture variations and long sequences, outperforming baselines on multiple datasets.

Contribution

It introduces a novel transformer-based model combining convolutional and attention mechanisms for improved sign language translation.

Findings

Achieved BLEU4 of 46.84% on CoL-SLTD

Achieved BLEU4 of 30.77% on PHOENIX14T

Demonstrated robustness in real-world scenarios

Abstract

Sign Language Translation (SLT) systems support hearing-impaired people communication by finding equivalences between signed and spoken languages. This task is however challenging due to multiple sign variations, complexity in language and inherent richness of expressions. Computational approaches have evidenced capabilities to support SLT. Nonetheless, these approaches remain limited to cover gestures variability and support long sequence translations. This paper introduces a Transformer-based architecture that encodes spatio-temporal motion gestures, preserving both local and long-range spatial information through the use of multiple convolutional and attention mechanisms. The proposed approach was validated on the Colombian Sign Language Translation Dataset (CoL-SLTD) outperforming baseline approaches, and achieving a BLEU4 of 46.84%. Additionally, the proposed approach was validated on the RWTH-PHOENIX-Weather-2014T (PHOENIX14T), achieving a BLEU4 score of 30.77%, demonstrating its robustness and effectiveness in handling real-world variations