Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition

TL;DR

This paper introduces a self-supervised contrastive learning framework that leverages spatial-temporal consistency, multi-granularity features, and modality interactions to improve sign language recognition accuracy.

Contribution

It proposes a novel contrastive learning approach that exploits spatial-temporal cues and modality interactions for richer sign language representations.

Findings

Achieves state-of-the-art results on four benchmarks.

Effectively encodes fine-grained hand and coarse-trunk features.

Utilizes motion and joint modality interactions for enhanced learning.

Abstract

Recently, there have been efforts to improve the performance in sign language recognition by designing self-supervised learning methods. However, these methods capture limited information from sign pose data in a frame-wise learning manner, leading to sub-optimal solutions. To this end, we propose a simple yet effective self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency from two distinct perspectives and learn instance discriminative representation for sign language recognition. On one hand, since the semantics of sign language are expressed by the cooperation of fine-grained hands and coarse-grained trunks, we utilize both granularity information and encode them into latent spaces. The consistency between hand and trunk features is constrained to encourage learning consistent representation of instance samples. On the other hand, inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Additionally, we further bridge the interaction between the embedding spaces of both modalities, facilitating bidirectional knowledge transfer to enhance sign language representation. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin. The source code is publicly available at https://github.com/sakura/Code.