Ham2Pose: Animating Sign Language Notation into Pose Sequences

TL;DR

This paper introduces Ham2Pose, a transformer-based method for animating sign language notation into pose sequences, offering a universal, data-efficient solution validated with a new distance metric on large-scale datasets.

Contribution

We propose the first universal method for translating HamNoSys notation into sign language pose sequences using transformers and weak supervision, along with a novel pose sequence distance measure.

Findings

Successfully animates HamNoSys notation into pose sequences

Outperforms existing distance measures in accuracy

Validated on large-scale Sign language dataset AUTSL

Abstract

Translating spoken languages into Sign languages is necessary for open communication between the hearing and hearing-impaired communities. To achieve this goal, we propose the first method for animating a text written in HamNoSys, a lexical Sign language notation, into signed pose sequences. As HamNoSys is universal, our proposed method offers a generic solution invariant to the target Sign language. Our method gradually generates pose predictions using transformer encoders that create meaningful representations of the text and poses while considering their spatial and temporal information. We use weak supervision for the training process and show that our method succeeds in learning from partial and inaccurate data. Additionally, we offer a new distance measurement for pose sequences, normalized Dynamic Time Warping (nDTW), based on DTW over normalized keypoints trajectories, and validate its correctness using AUTSL, a large-scale Sign language dataset. We show that it measures the distance between pose sequences more accurately than existing measurements and use it to assess the quality of our generated pose sequences. Code for the data pre-processing, the model, and the distance measurement is publicly released for future research.