# ADAT novel time-series-aware adaptive transformer architecture for sign language translation

## TL;DR

This paper introduces ADAT, a new Transformer-based model for sign language translation that improves accuracy and training efficiency by better capturing temporal dependencies.

## Contribution

ADAT introduces an adaptive Transformer architecture with log-sparse attention and dual-stream processing for efficient sign language translation.

## Key findings

- ADAT improves BLEU-4 scores by at least 0.1% in sign-to-gloss-to-text translation.
- ADAT reduces training time by an average of 21% across datasets.
- ADAT achieves 0.5% gains in BLEU-4 for sign-to-text translation over baselines.

## Abstract

Current sign language machine translation systems rely on recognizing hand movements, facial expressions, and body postures, and natural language processing, to convert signs into text. While recent approaches use Transformer architectures to model long-range dependencies via positional encoding, they lack accuracy in recognizing fine-grained, short-range temporal dependencies between gestures captured at high frame rates. Moreover, their quadratic attention complexity leads to inefficient training. To mitigate these issues, we introduce ADAT, an Adaptive Transformer architecture that combines convolutional feature extraction, log-sparse self-attention, and an adaptive gating mechanism to efficiently model both short- and long-range temporal dependencies in sign language sequences. We evaluate ADAT on three datasets: the benchmark RWTH-PHOENIX-Weather-2014 (PHOENIX14T), the ISL-CSLTR, and the newly introduced MedASL, a medical-domain American Sign Language corpus. In sign-to-gloss-to-text translation, ADAT outperforms the state-of-the-art baselines, improving BLEU-4 by at least 0.1% and reducing training time by an average of 21% across datasets. In sign-to-text translation, ADAT consistently surpasses transformer-based encoder-decoder baselines, achieving a minimum of 0.5% gains in BLEU-4 and an average training speedup of 21.8% across datasets. Compared to the encoder-only and decoder-only baselines in sign-to-text, ADAT is at least 0.7% more accurate, despite being up to 12.1% slower due to its dual-stream structure.

The online version contains supplementary material available at 10.1038/s41598-026-36293-9.