Stable Signer: Hierarchical Sign Language Generative Model

TL;DR

This paper introduces Stable Signer, a hierarchical end-to-end model for sign language video generation that simplifies the process, improves accuracy, and achieves significant performance gains over existing methods.

Contribution

The paper proposes a novel hierarchical sign language generation framework with a new understanding linker and gesture rendering expert, streamlining the process and enhancing quality.

Findings

Performance improved by 48.6% over SOTA methods

Introduces SLUL for better text understanding in sign language generation

Develops SAGM Loss for improved training of understanding linkers

Abstract

Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.