Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing

TL;DR

This paper introduces Gloss2Text, a novel approach leveraging large language models, data augmentation, and a new label-smoothing loss to improve sign language gloss translation accuracy from videos.

Contribution

It proposes a new gloss-to-text translation method that significantly enhances performance by exploiting LLMs, data augmentation, and a novel label-smoothing loss function.

Findings

Outperforms state-of-the-art on PHOENIX Weather 2014T dataset

Demonstrates the effectiveness of label smoothing in gloss translation

Shows significant improvement over previous methods

Abstract

Sign language translation from video to spoken text presents unique challenges owing to the distinct grammar, expression nuances, and high variation of visual appearance across different speakers and contexts. The intermediate gloss annotations of videos aim to guide the translation process. In our work, we focus on {\em Gloss2Text} translation stage and propose several advances by leveraging pre-trained large language models (LLMs), data augmentation, and novel label-smoothing loss function exploiting gloss translation ambiguities improving significantly the performance of state-of-the-art approaches. Through extensive experiments and ablation studies on the PHOENIX Weather 2014T dataset, our approach surpasses state-of-the-art performance in {\em Gloss2Text} translation, indicating its efficacy in addressing sign language translation and suggesting promising avenues for future research and development.