Improving Gloss-free Sign Language Translation by Reducing Representation Density

TL;DR

This paper introduces SignCL, a contrastive learning strategy that reduces representation density in gloss-free sign language translation, significantly improving translation accuracy without increasing model size.

Contribution

The paper proposes a novel contrastive learning method, SignCL, to address the representation density problem in gloss-free SLT, enhancing discriminative feature learning and translation performance.

Findings

SignCL reduces representation density in feature space.

SignCL improves BLEU scores by 39% and 46% on CSL-Daily dataset.

SignCL outperforms Sign2GPT with fewer parameters.

Abstract

Gloss-free sign language translation (SLT) aims to develop well-performing SLT systems with no requirement for the costly gloss annotations, but currently still lags behind gloss-based approaches significantly. In this paper, we identify a representation density problem that could be a bottleneck in restricting the performance of gloss-free SLT. Specifically, the representation density problem describes that the visual representations of semantically distinct sign gestures tend to be closely packed together in feature space, which makes gloss-free methods struggle with distinguishing different sign gestures and suffer from a sharp performance drop. To address the representation density problem, we introduce a simple but effective contrastive learning strategy, namely SignCL, which encourages gloss-free models to learn more discriminative feature representation in a self-supervised manner. Our experiments demonstrate that the proposed SignCL can significantly reduce the representation density and improve performance across various translation frameworks. Specifically, SignCL achieves a significant improvement in BLEU score for the Sign Language Transformer and GFSLT-VLP on the CSL-Daily dataset by 39% and 46%, respectively, without any increase of model parameters. Compared to Sign2GPT, a state-of-the-art method based on large-scale pre-trained vision and language models, SignCL achieves better performance with only 35% of its parameters. Implementation and Checkpoints are available at https://github.com/JinhuiYE/SignCL.