Beyond a Single Reference: Training and Evaluation with Paraphrases in Sign Language Translation

TL;DR

This paper explores using large language models to generate paraphrased references for sign language translation, improving evaluation metrics and aligning better with human judgment, while also examining the effects on training.

Contribution

It introduces BLEUpara, a new evaluation metric using multiple paraphrased references, and analyzes the impact of paraphrases on SLT training and evaluation.

Findings

Paraphrases improve evaluation scores when used during testing.

Naive incorporation of paraphrases in training does not enhance performance.

BLEUpara correlates more strongly with human judgments.

Abstract

Most Sign Language Translation (SLT) corpora pair each signed utterance with a single written-language reference, despite the highly non-isomorphic relationship between sign and spoken languages, where multiple translations can be equally valid. This limitation constrains both model training and evaluation, particularly for n-gram-based metrics such as BLEU. In this work, we investigate the use of Large Language Models to automatically generate paraphrased variants of written-language translations as synthetic alternative references for SLT. First, we compare multiple paraphrasing strategies and models using an adapted ParaScore metric. Second, we study the impact of paraphrases on both training and evaluation of the pose-based T5 model on the YouTubeASL and How2Sign datasets. Our results show that naively incorporating paraphrases during training does not improve translation performance and can even be detrimental. In contrast, using paraphrases during evaluation leads to higher automatic scores and better alignment with human judgments. To formalize this observation, we introduce BLEUpara, an extension of BLEU that evaluates translations against multiple paraphrased references. Human evaluation confirms that BLEUpara correlates more strongly with perceived translation quality. We release all generated paraphrases, generation and evaluation code to support reproducible and more reliable evaluation of SLT systems.