Pose-based Sign Language Recognition using GCN and BERT

TL;DR

This paper introduces a novel pose-based sign language recognition model that separately captures spatial and temporal information using GCN and BERT, achieving significant accuracy improvements on the WLASL dataset.

Contribution

The paper presents a new architecture that explicitly models spatial and temporal dependencies separately and fuses them, advancing pose-based WSLR methods.

Findings

Outperforms state-of-the-art pose-based methods by up to 5% accuracy

Uses GCN for spatial interactions and BERT for temporal dependencies

Achieves significant improvements on the WLASL dataset

Abstract

Sign language recognition (SLR) plays a crucial role in bridging the communication gap between the hearing and vocally impaired community and the rest of the society. Word-level sign language recognition (WSLR) is the first important step towards understanding and interpreting sign language. However, recognizing signs from videos is a challenging task as the meaning of a word depends on a combination of subtle body motions, hand configurations, and other movements. Recent pose-based architectures for WSLR either model both the spatial and temporal dependencies among the poses in different frames simultaneously or only model the temporal information without fully utilizing the spatial information. We tackle the problem of WSLR using a novel pose-based approach, which captures spatial and temporal information separately and performs late fusion. Our proposed architecture explicitly captures the spatial interactions in the video using a Graph Convolutional Network (GCN). The temporal dependencies between the frames are captured using Bidirectional Encoder Representations from Transformers (BERT). Experimental results on WLASL, a standard word-level sign language recognition dataset show that our model significantly outperforms the state-of-the-art on pose-based methods by achieving an improvement in the prediction accuracy by up to 5%.