Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues

TL;DR

This paper introduces a novel sign language translation framework that leverages contextual cues such as captions, previous translations, and pseudo-glosses, significantly improving translation accuracy by integrating visual and textual information with a large language model.

Contribution

The paper presents a new multi-modal translation approach that incorporates various contextual cues into a large language model for improved sign language translation accuracy.

Findings

Enhanced translation quality on BOBSL dataset

Significant improvement over previous methods

Effective generalization to How2Sign dataset

Abstract

Our objective is to translate continuous sign language into spoken language text. Inspired by the way human interpreters rely on context for accurate translation, we incorporate additional contextual cues together with the signing video, into a new translation framework. Specifically, besides visual sign recognition features that encode the input video, we integrate complementary textual information from (i) captions describing the background show, (ii) translation of previous sentences, as well as (iii) pseudo-glosses transcribing the signing. These are automatically extracted and inputted along with the visual features to a pre-trained large language model (LLM), which we fine-tune to generate spoken language translations in text form. Through extensive ablation studies, we show the positive contribution of each input cue to the translation performance. We train and evaluate our approach on BOBSL -- the largest British Sign Language dataset currently available. We show that our contextual approach significantly enhances the quality of the translations compared to previously reported results on BOBSL, and also to state-of-the-art methods that we implement as baselines. Furthermore, we demonstrate the generality of our approach by applying it also to How2Sign, an American Sign Language dataset, and achieve competitive results.