Teach Me Sign: Stepwise Prompting LLM for Sign Language Production

TL;DR

This paper introduces TEAch Me Sign (TEAM-Sign), a method that fine-tunes large language models with stepwise prompting to improve sign language generation by treating it as a natural language, leveraging reasoning and knowledge.

Contribution

The paper presents a novel approach to sign language generation by fine-tuning LLMs with stepwise prompting, effectively capturing sign language rules and knowledge.

Findings

Effective sign language generation on How2Sign dataset

Improved alignment of sign and spoken language distributions

Leveraging LLM reasoning for sign language tasks

Abstract

Large language models, with their strong reasoning ability and rich knowledge, have brought revolution to many tasks of AI, but their impact on sign language generation remains limited due to its complexity and unique rules. In this paper, we propose TEAch Me Sign (TEAM-Sign), treating sign language as another natural language. By fine-tuning an LLM, we enable it to learn the correspondence between text and sign language, and facilitate generation. Considering the differences between sign and spoken language, we employ a stepwise prompting strategy to extract the inherent sign language knowledge within the LLM, thereby supporting the learning and generation process. Experimental results on How2Sign and Phoenix14T datasets demonstrate that our approach effectively leverages both the sign language knowledge and reasoning capabilities of LLM to align the different distribution and grammatical rules between sign and spoken language.