A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production

TL;DR

This paper introduces UniGloR, a spatio-temporal representation framework that replaces traditional glosses in sign language translation and production, capturing dynamic features for improved performance.

Contribution

We propose UniGloR, a novel dense spatio-temporal representation derived from keypoints, addressing gloss annotation limitations and enhancing sign language processing tasks.

Findings

Outperforms previous methods on PHOENIX14T and How2Sign datasets.

Effectively captures sign language dynamics without gloss annotations.

Matches or exceeds state-of-the-art performance in SLT and SLP.

Abstract

This work addresses the challenges associated with the use of glosses in both Sign Language Translation (SLT) and Sign Language Production (SLP). While glosses have long been used as a bridge between sign language and spoken language, they come with two major limitations that impede the advancement of sign language systems. First, annotating the glosses is a labor-intensive and time-consuming process, which limits the scalability of datasets. Second, the glosses oversimplify sign language by stripping away its spatio-temporal dynamics, reducing complex signs to basic labels and missing the subtle movements essential for precise interpretation. To address these limitations, we introduce Universal Gloss-level Representation (UniGloR), a framework designed to capture the spatio-temporal features inherent in sign language, providing a more dynamic and detailed alternative to the use of the glosses. The core idea of UniGloR is simple yet effective: We derive dense spatio-temporal representations from sign keypoint sequences using self-supervised learning and seamlessly integrate them into SLT and SLP tasks. Our experiments in a keypoint-based setting demonstrate that UniGloR either outperforms or matches the performance of previous SLT and SLP methods on two widely-used datasets: PHOENIX14T and How2Sign.