Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network

TL;DR

This paper introduces a hierarchical spatio-temporal graph neural network for sign language translation, capturing complex visual-manual features to improve translation accuracy over traditional sequence models.

Contribution

The paper proposes a novel hierarchical graph-based neural network that models multi-level visual features of sign language for improved translation performance.

Findings

Effective in capturing multi-level sign language features

Outperforms existing methods on benchmark datasets

Demonstrates the importance of hierarchical graph modeling

Abstract

Sign language translation (SLT), which generates text in a spoken language from visual content in a sign language, is important to assist the hard-of-hearing community for their communications. Inspired by neural machine translation (NMT), most existing SLT studies adopted a general sequence to sequence learning strategy. However, SLT is significantly different from general NMT tasks since sign languages convey messages through multiple visual-manual aspects. Therefore, in this paper, these unique characteristics of sign languages are formulated as hierarchical spatio-temporal graph representations, including high-level and fine-level graphs of which a vertex characterizes a specified body part and an edge represents their interactions. Particularly, high-level graphs represent the patterns in the regions such as hands and face, and fine-level graphs consider the joints of hands and landmarks of facial regions. To learn these graph patterns, a novel deep learning architecture, namely hierarchical spatio-temporal graph neural network (HST-GNN), is proposed. Graph convolutions and graph self-attentions with neighborhood context are proposed to characterize both the local and the global graph properties. Experimental results on benchmark datasets demonstrated the effectiveness of the proposed method.