Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation

TL;DR

This paper introduces a novel gloss-free pseudo gloss generation method for sign language translation that leverages large language models and weakly supervised learning to improve alignment and performance without relying on costly gloss annotations.

Contribution

It proposes a new framework that generates pseudo glosses from spoken language text using LLMs, enhancing sign language translation without expert-annotated gloss labels.

Findings

Outperforms previous gloss-free methods on SLT benchmarks.

Achieves competitive results compared to gloss-based approaches.

Uses weakly supervised learning for better alignment of pseudo glosses.

Abstract

Sign Language Translation (SLT) aims to map sign language videos to spoken language text. A common approach relies on gloss annotations as an intermediate representation, decomposing SLT into two sub-tasks: video-to-gloss recognition and gloss-to-text translation. While effective, this paradigm depends on expert-annotated gloss labels, which are costly and rarely available in existing datasets, limiting its scalability. To address this challenge, we propose a gloss-free pseudo gloss generation framework that eliminates the need for human-annotated glosses while preserving the structured intermediate representation. Specifically, we prompt a Large Language Model (LLM) with a few example text-gloss pairs using in-context learning to produce draft sign glosses from spoken language text. To enhance the correspondence between LLM-generated pseudo glosses and the sign sequences in video, we correct the ordering in the pseudo glosses for better alignment via a weakly supervised learning process. This reordering facilitates the incorporation of auxiliary alignment objectives, and allows for the use of efficient supervision via a Connectionist Temporal Classification (CTC) loss. We train our SLT mode, which consists of a vision encoder and a translator, through a three-stage pipeline, which progressively narrows the modality gap between sign language and spoken language. Despite its simplicity, our approach outperforms previous state-of-the-art gloss-free frameworks on two SLT benchmarks and achieves competitive results compared to gloss-based methods.