Sign Stitching: A Novel Approach to Sign Language Production

TL;DR

This paper introduces Sign Stitching, a novel 7-step method for producing natural, expressive sign language sequences from text, leveraging dictionary examples and a SignGAN model for photo-realistic signing.

Contribution

It presents a new 7-step sign stitching approach combined with SignGAN to improve naturalness and expressiveness in text-to-sign language production.

Findings

Achieved state-of-the-art performance across multiple datasets.

Produced more natural and expressive sign sequences.

Enhanced sign language production quality significantly.

Abstract

Sign Language Production (SLP) is a challenging task, given the limited resources available and the inherent diversity within sign data. As a result, previous works have suffered from the problem of regression to the mean, leading to under-articulated and incomprehensible signing. In this paper, we propose using dictionary examples to create expressive sign language sequences. However, simply concatenating the signs would create robotic and unnatural sequences. Therefore, we present a 7-step approach to effectively stitch the signs together. First, by normalising each sign into a canonical pose, cropping and stitching we create a continuous sequence. Then by applying filtering in the frequency domain and resampling each sign we create cohesive natural sequences, that mimic the prosody found in the original data. We leverage the SignGAN model to map the output to a photo-realistic signer and present a complete Text-to-Sign (T2S) SLP pipeline. Our evaluation demonstrates the effectiveness of this approach, showcasing state-of-the-art performance across all datasets.