Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives

TL;DR

This paper introduces a novel sign language production method using a Mixture of Motion Primitives, improving translation accuracy and animation realism by splitting the task into translation and animation sub-tasks with a joint training approach.

Contribution

It proposes a new architecture combining a progressive transformer with a Mixture of Motion Primitives for sign language animation, outperforming existing methods in translation and animation quality.

Findings

Outperforms baselines in user evaluations

Achieves 11% improvement in back translation performance

Demonstrates stronger spoken-to-sign translation than gloss-to-sign

Abstract

It is common practice to represent spoken languages at their phonetic level. However, for sign languages, this implies breaking motion into its constituent motion primitives. Avatar based Sign Language Production (SLP) has traditionally done just this, building up animation from sequences of hand motions, shapes and facial expressions. However, more recent deep learning based solutions to SLP have tackled the problem using a single network that estimates the full skeletal structure. We propose splitting the SLP task into two distinct jointly-trained sub-tasks. The first translation sub-task translates from spoken language to a latent sign language representation, with gloss supervision. Subsequently, the animation sub-task aims to produce expressive sign language sequences that closely resemble the learnt spatio-temporal representation. Using a progressive transformer for the translation sub-task, we propose a novel Mixture of Motion Primitives (MoMP) architecture for sign language animation. A set of distinct motion primitives are learnt during training, that can be temporally combined at inference to animate continuous sign language sequences. We evaluate on the challenging RWTH-PHOENIX-Weather-2014T(PHOENIX14T) dataset, presenting extensive ablation studies and showing that MoMP outperforms baselines in user evaluations. We achieve state-of-the-art back translation performance with an 11% improvement over competing results. Importantly, and for the first time, we showcase stronger performance for a full translation pipeline going from spoken language to sign, than from gloss to sign.