SONAR-SLT: Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision

TL;DR

This paper introduces SONAR-SLT, a multilingual sign language translation method using language-agnostic sentence embeddings and data augmentation, improving cross-language generalization and robustness especially in low-resource scenarios.

Contribution

It proposes a novel multilingual SLT approach with language-agnostic embeddings and coupled augmentation, enhancing scalability and performance over previous methods.

Findings

Consistent BLEURT improvements over text-only supervision.

Significant gains in low-resource language settings.

Enhanced model robustness through data augmentation.

Abstract

Sign language translation (SLT) is typically trained with text in a single spoken language, which limits scalability and cross-language generalization. Earlier approaches have replaced gloss supervision with text-based sentence embeddings, but up to now, these remain tied to a specific language and modality. In contrast, here we employ language-agnostic, multimodal embeddings trained on text and speech from multiple languages to supervise SLT, enabling direct multilingual translation. To address data scarcity, we propose a coupled augmentation method that combines multilingual target augmentations (i.e. translations into many languages) with video-level perturbations, improving model robustness. Experiments show consistent BLEURT gains over text-only sentence embedding supervision, with larger improvements in low-resource settings. Our results demonstrate that language-agnostic embedding supervision, combined with coupled augmentation, provides a scalable and semantically robust alternative to traditional SLT training.