SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition

TL;DR

SignMAE introduces a segmentation-driven self-supervised learning approach that enhances fine-grained sign language recognition by focusing on key body parts, achieving state-of-the-art results with fewer input frames.

Contribution

The paper presents a novel segmentation-based masking pretraining method tailored for sign language recognition, improving fine-grained cue capture over generic pretraining.

Findings

Achieves state-of-the-art accuracy on WLASL, NMFs-CSL, and Slovo datasets.

Improves per-instance and per-class Top-1 accuracy.

Uses fewer input frames and modalities than comparable methods.

Abstract

Subtle hand differences make sign language recognition challenging, yet many existing methods rely on encoders pretrained on generic action datasets that poorly capture such fine-grained cues. We propose a self-supervised pretraining method for sign language recognition that uses segmentation-based masking to adapt to the presence and motion of key body parts, rather than treating hand poses as static visual tokens. The resulting mask-and-reconstruct objective improves fine-grained sign representation learning. On WLASL, NMFs-CSL, and Slovo, our encoder achieves state-of-the-art performance, improving per-instance and per-class Top-1 accuracy while using fewer input frames and modalities than comparable encoders.