TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation

TL;DR

TSPNet introduces a hierarchical feature learning approach using a temporal semantic pyramid to improve sign language translation by capturing multi-granularity temporal features and semantic context, leading to significant performance gains.

Contribution

The paper proposes a novel hierarchical sign video feature learning method with a temporal semantic pyramid that effectively models multi-scale temporal semantics without explicit segmentation.

Findings

Outperforms state-of-the-art on SLT dataset with BLEU score of 13.41

Achieves significant improvements in ROUGE score to 34.96

Demonstrates the effectiveness of multi-granularity temporal features

Abstract

Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with no clear boundaries in between. Existing SLT models usually represent sign visual features in a frame-wise manner so as to avoid needing to explicitly segmenting the videos into isolated signs. However, these methods neglect the temporal information of signs and lead to substantial ambiguity in translation. In this paper, we explore the temporal semantic structures of signvideos to learn more discriminative features. To this end, we first present a novel sign video segment representation which takes into account multiple temporal granularities, thus alleviating the need for accurate video segmentation. Taking advantage of the proposed segment representation, we develop a novel hierarchical sign video feature learning method via a temporal semantic pyramid network, called TSPNet. Specifically, TSPNet introduces an inter-scale attention to evaluate and enhance local semantic consistency of sign segments and an intra-scale attention to resolve semantic ambiguity by using non-local video context. Experiments show that our TSPNet outperforms the state-of-the-art with significant improvements on the BLEU score (from 9.58 to 13.41) and ROUGE score (from 31.80 to 34.96)on the largest commonly-used SLT dataset. Our implementation is available at https://github.com/verashira/TSPNet.