Scaling up Multimodal Pre-training for Sign Language Understanding

TL;DR

This paper introduces a large-scale multimodal pre-training approach to improve sign language understanding across recognition, translation, and retrieval tasks, addressing the challenge of learning effective representations of sign language videos.

Contribution

It proposes a unified multimodal pre-training framework that enhances performance across multiple sign language understanding tasks, a novel approach in this domain.

Findings

Improved accuracy in sign language recognition and translation tasks.

Enhanced retrieval performance with the proposed pre-training model.

Demonstrated generalization across diverse SLU tasks.

Abstract

Sign language serves as the primary meaning of communication for the deaf-mute community. Different from spoken language, it commonly conveys information by the collaboration of manual features, i.e., hand gestures and body movements, and non-manual features, i.e., facial expressions and mouth cues. To facilitate communication between the deaf-mute and hearing people, a series of sign language understanding (SLU) tasks have been studied in recent years, including isolated/continuous sign language recognition (ISLR/CSLR), gloss-free sign language translation (GF-SLT) and sign language retrieval (SL-RT). Sign language recognition and translation aims to understand the semantic meaning conveyed by sign languages from gloss-level and sentence-level, respectively. In contrast, SL-RT focuses on retrieving sign videos or corresponding texts from a closed-set under the query-by-example search paradigm. These tasks investigate sign language topics from diverse perspectives and raise challenges in learning effective representation of sign language videos. To advance the development of sign language understanding, exploring a generalized model that is applicable across various SLU tasks is a profound research direction.