Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios

TL;DR

This paper introduces AulSign, a novel approach leveraging large language models with dynamic prompting and sign association to improve sign language translation in low-resource scenarios, enhancing accessibility for underrepresented communities.

Contribution

The paper presents AulSign, a new method that adapts LLMs for sign language translation using natural language descriptions of signs, addressing data scarcity issues.

Findings

AulSign outperforms existing models in low-data environments.

The method effectively associates signs with natural language descriptions.

Results demonstrate improved translation accuracy on benchmark datasets.

Abstract

Translating natural languages into sign languages is a highly complex and underexplored task. Despite growing interest in accessibility and inclusivity, the development of robust translation systems remains hindered by the limited availability of parallel corpora which align natural language with sign language data. Existing methods often struggle to generalize in these data-scarce environments, as the few datasets available are typically domain-specific, lack standardization, or fail to capture the full linguistic richness of sign languages. To address this limitation, we propose Advanced Use of LLMs for Sign Language Translation (AulSign), a novel method that leverages Large Language Models via dynamic prompting and in-context learning with sample selection and subsequent sign association. Despite their impressive abilities in processing text, LLMs lack intrinsic knowledge of sign languages; therefore, they are unable to natively perform this kind of translation. To overcome this limitation, we associate the signs with compact descriptions in natural language and instruct the model to use them. We evaluate our method on both English and Italian languages using SignBank+, a recognized benchmark in the field, as well as the Italian LaCAM CNR-ISTC dataset. We demonstrate superior performance compared to state-of-the-art models in low-data scenario. Our findings demonstrate the effectiveness of AulSign, with the potential to enhance accessibility and inclusivity in communication technologies for underrepresented linguistic communities.