Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation

TL;DR

Sign2GPT introduces a novel framework leveraging large-scale pretrained models and lightweight adapters for effective gloss-free sign language translation, addressing data scarcity and computational challenges.

Contribution

The paper proposes a new sign language translation method using pretrained vision and language models with adapters and a novel pretraining strategy from pseudo-glosses, without requiring gloss annotations.

Findings

Achieved significant improvements over state-of-the-art in gloss-free translation.

Effectively learned sign representations from pseudo-glosses without gloss order annotations.

Demonstrated robustness across two benchmark datasets.

Abstract

Automatic Sign Language Translation requires the integration of both computer vision and natural language processing to effectively bridge the communication gap between sign and spoken languages. However, the deficiency in large-scale training data to support sign language translation means we need to leverage resources from spoken language. We introduce, Sign2GPT, a novel framework for sign language translation that utilizes large-scale pretrained vision and language models via lightweight adapters for gloss-free sign language translation. The lightweight adapters are crucial for sign language translation, due to the constraints imposed by limited dataset sizes and the computational requirements when training with long sign videos. We also propose a novel pretraining strategy that directs our encoder to learn sign representations from automatically extracted pseudo-glosses without requiring gloss order information or annotations. We evaluate our approach on two public benchmark sign language translation datasets, namely RWTH-PHOENIX-Weather 2014T and CSL-Daily, and improve on state-of-the-art gloss-free translation performance with a significant margin.