Towards Compact Sign Language Translation: Frame Rate and Model Size Trade-offs

TL;DR

This paper presents a compact sign language translation system that balances frame rate and model size, achieving high efficiency and competitive accuracy with significantly smaller models.

Contribution

A novel lightweight SLT pipeline combining skeletal pose extraction with a small language model, optimizing frame rate for efficiency without substantial accuracy loss.

Findings

Halves sequence length at 12 fps, reducing computational complexity by 75%.

Achieves roughly 3x smaller size than prior T5-base systems.

Maintains competitive BLEU-4 scores with reduced model size.

Abstract

Sign Language Translation (SLT) converts sign language videos into spoken-language text, bridging communication between Deaf and hearing communities. Current gloss-free approaches rely on large encoder-decoder models, limiting deployment. We propose a compact 77M-parameter pipeline that couples MMPose skeletal pose extraction with a single linear projection into T5-small. By varying the input frame rate, we expose a practical efficiency trade-off: at 12 fps the model halves its sequence length, achieving a 75% reduction in encoder quadratic self-attention computational complexity while incurring only a modest BLEU-4 drop (9.53 vs. 10.06 at 24 fps on How2Sign). Our system is roughly 3x smaller than prior T5-base systems, demonstrating that a lightweight architecture can remain competitive without hierarchical encoders or large-scale models.