Towards Privacy-Aware Sign Language Translation at Scale

TL;DR

This paper presents a privacy-aware sign language translation framework that uses self-supervised pretraining on anonymized videos, achieving state-of-the-art results while addressing data scarcity and privacy concerns.

Contribution

It introduces SSVP-SLT, a two-stage framework combining self-supervised pretraining on anonymized videos with supervised finetuning, advancing privacy-aware large-scale sign language translation.

Findings

Achieves over 3 BLEU-4 improvement on How2Sign dataset

Demonstrates effectiveness of self-supervised pretraining for SLT

Discusses privacy benefits of facial obfuscation techniques

Abstract

A major impediment to the advancement of sign language translation (SLT) is data scarcity. Much of the sign language data currently available on the web cannot be used for training supervised models due to the lack of aligned captions. Furthermore, scaling SLT using large-scale web-scraped datasets bears privacy risks due to the presence of biometric information, which the responsible development of SLT technologies should account for. In this work, we propose a two-stage framework for privacy-aware SLT at scale that addresses both of these issues. We introduce SSVP-SLT, which leverages self-supervised video pretraining on anonymized and unannotated videos, followed by supervised SLT finetuning on a curated parallel dataset. SSVP-SLT achieves state-of-the-art finetuned and zero-shot gloss-free SLT performance on the How2Sign dataset, outperforming the strongest respective baselines by over 3 BLEU-4. Based on controlled experiments, we further discuss the advantages and limitations of self-supervised pretraining and anonymization via facial obfuscation for SLT.