SignLLM: Sign Language Production Large Language Models

TL;DR

SignLLM is a multilingual large language model for sign language production that introduces novel modes and reinforcement learning techniques, achieving state-of-the-art results across eight sign languages.

Contribution

The paper presents SignLLM, a new multilingual sign language production model with two innovative modes and a reinforcement learning framework, along with a comprehensive dataset for training.

Findings

Achieves state-of-the-art performance on SLP tasks

Introduces two novel multilingual SLP modes

Develops a new RL-based training approach

Abstract

In this paper, we propose SignLLM, a multilingual Sign Language Production (SLP) large language model, which includes two novel multilingual SLP modes MLSF and Prompt2LangGloss that allow sign language gestures generation from query texts input and question-style prompts input respectively. Both modes can use a new RL loss based on reinforcement learning and a new RL module named Priority Learning Channel. These RL components can accelerate the training by enhancing the model's capability to sample high-quality data. To train SignLLM, we introduce Prompt2Sign, a comprehensive multilingual sign language dataset, which builds from public data, including American Sign Language (ASL) and seven others. This dataset standardizes information by extracting pose information from sign language videos into a unified compressed format. We extensively evaluate SignLLM, demonstrating that our model achieves state-of-the-art performance on SLP tasks across eight sign languages.