Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment

TL;DR

This paper introduces a unified, scalable model for sign language translation and subtitle alignment that leverages multilingual pretraining and a novel architecture to achieve state-of-the-art results and cross-linguistic generalization.

Contribution

The paper presents a new multi-task model combining visual feature extraction, a Sliding Perceiver network, and scalable training for sign language translation and alignment, with multilingual pretraining.

Findings

Achieves state-of-the-art results on BSL dataset for SLT and SSA.

Demonstrates robust zero-shot generalization to ASL.

Effective multilingual pretraining enhances cross-linguistic performance.

Abstract

Our aim is to develop a unified model for sign language understanding, that performs sign language translation (SLT) and sign-subtitle alignment (SSA). Together, these two tasks enable the conversion of continuous signing videos into spoken language text and also the temporal alignment of signing with subtitles -- both essential for practical communication, large-scale corpus construction, and educational applications. To achieve this, our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images while preserving signer privacy; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings to bridge the vision-text gap; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA, reinforcing both linguistic and temporal alignment. To promote cross-linguistic generalisation, we pretrain our model on large-scale sign-text corpora covering British Sign Language (BSL) and American Sign Language (ASL) from the BOBSL and YouTube-SL-25 datasets. With this multilingual pretraining and strong model design, we achieve state-of-the-art results on the challenging BOBSL (BSL) dataset for both SLT and SSA. Our model also demonstrates robust zero-shot generalisation and finetuned SLT performance on How2Sign (ASL), highlighting the potential of scalable translation across different sign languages.