HandCraft: Dynamic Sign Generation for Synthetic Data Augmentation

TL;DR

This paper introduces a lightweight sign generation model and synthetic data pretraining approach that significantly improve Sign Language Recognition accuracy, outperforming traditional methods and establishing new state-of-the-art results across multiple datasets.

Contribution

The paper presents a novel, computationally efficient sign generation model based on CMLPe and a synthetic data pretraining method that enhances SLR performance.

Findings

Synthetic pretraining outperforms traditional augmentation in some cases.

Combining synthetic data with augmentation yields complementary benefits.

Achieves state-of-the-art results on LSFB and DiSPLaY datasets.

Abstract

Sign Language Recognition (SLR) models face significant performance limitations due to insufficient training data availability. In this article, we address the challenge of limited data in SLR by introducing a novel and lightweight sign generation model based on CMLPe. This model, coupled with a synthetic data pretraining approach, consistently improves recognition accuracy, establishing new state-of-the-art results for the LSFB and DiSPLaY datasets using our Mamba-SL and Transformer-SL classifiers. Our findings reveal that synthetic data pretraining outperforms traditional augmentation methods in some cases and yields complementary benefits when implemented alongside them. Our approach democratizes sign generation and synthetic data pretraining for SLR by providing computationally efficient methods that achieve significant performance improvements across diverse datasets.