Example-Based Machine Translation from Text to a Hierarchical Representation of Sign Language

TL;DR

This paper introduces a novel Text-to-Sign Language translation method that leverages hierarchical representations and similarity detection to generate expressive, avatar-compatible sign language translations from text.

Contribution

It presents an original recursive algorithm for translating text into hierarchical sign language representations using a domain-specific corpus, addressing data scarcity and preserving linguistic structure.

Findings

Effective recursive translation algorithm demonstrated

Generated translations show potential for expressiveness and idiomatic language

Identified limitations and future evaluation prospects

Abstract

This article presents an original method for Text-to-Sign Translation. It compensates data scarcity using a domain-specific parallel corpus of alignments between text and hierarchical formal descriptions of Sign Language videos in AZee. Based on the detection of similarities present in the source text, the proposed algorithm recursively exploits matches and substitutions of aligned segments to build multiple candidate translations for a novel statement. This helps preserving Sign Language structures as much as possible before falling back on literal translations too quickly, in a generative way. The resulting translations are in the form of AZee expressions, designed to be used as input to avatar synthesis systems. We present a test set tailored to showcase its potential for expressiveness and generation of idiomatic target language, and observed limitations. This work finally opens prospects on how to evaluate translation and linguistic aspects, such as accuracy and grammatical fluency.