GLOS: Sign Language Generation with Temporally Aligned Gloss-Level Conditioning

TL;DR

GLOS introduces a sign language generation framework that uses temporally aligned gloss-level conditioning and a new fusion module to improve lexical order and semantic accuracy in generated signs.

Contribution

The paper presents a novel gloss-level conditioning approach with temporal alignment and a fusion module, enhancing sign language generation accuracy and control.

Findings

Outperforms prior methods on CSL-Daily and Phoenix-2014T datasets.

Generates signs with correct lexical order and high semantic accuracy.

Enables fine-grained control of signs through word-level semantics.

Abstract

Sign language generation (SLG), or text-to-sign generation, bridges the gap between signers and non-signers. Despite recent progress in SLG, existing methods still often suffer from incorrect lexical ordering and low semantic accuracy. This is primarily due to sentence-level condition, which encodes the entire sentence of the input text into a single feature vector as a condition for SLG. This approach fails to capture the temporal structure of sign language and lacks the granularity of word-level semantics, often leading to disordered sign sequences and ambiguous motions. To overcome these limitations, we propose GLOS, a sign language generation framework with temporally aligned gloss-level conditioning. First, we employ gloss-level conditions, which we define as sequences of gloss embeddings temporally aligned with the motion sequence. This enables the model to access both the temporal structure of sign language and word-level semantics at each timestep. As a result, this allows for fine-grained control of signs and better preservation of lexical order. Second, we introduce a condition fusion module, temporal alignment conditioning (TAC), to efficiently deliver the word-level semantic and temporal structure provided by the gloss-level condition to the corresponding motion timesteps. Our method, which is composed of gloss-level conditions and TAC, generates signs with correct lexical order and high semantic accuracy, outperforming prior methods on CSL-Daily and Phoenix-2014T.