Generative Sign-description Prompts with Multi-positive Contrastive Learning for Sign Language Recognition

TL;DR

This paper introduces GSP-MC, a novel approach that integrates generative large language models with contrastive learning to improve sign language recognition accuracy across multiple languages.

Contribution

It is the first to incorporate generative LLMs into SLR, using retrieval-augmented generation and multi-level text-skeleton alignment for enhanced recognition.

Findings

Achieved 97.1% accuracy on Chinese SLR500 dataset.

Achieved 97.07% accuracy on Turkish AUTSL dataset.

Demonstrated state-of-the-art performance and cross-lingual effectiveness.

Abstract

Sign language recognition (SLR) faces fundamental challenges in creating accurate annotations due to the inherent complexity of simultaneous manual and non-manual signals. To the best of our knowledge, this is the first work to integrate generative large language models (LLMs) into SLR tasks. We propose a novel Generative Sign-description Prompts Multi-positive Contrastive learning (GSP-MC) method that leverages retrieval-augmented generation (RAG) with domain-specific LLMs, incorporating multi-step prompt engineering and expert-validated sign language corpora to produce precise multipart descriptions. The GSP-MC method also employs a dual-encoder architecture to bidirectionally align hierarchical skeleton features with multiple text descriptions (global, synonym, and part level) through probabilistic matching. Our approach combines global and part-level losses, optimizing KL divergence to ensure robust alignment across all relevant text-skeleton pairs while capturing both sign-level semantics and detailed part dynamics. Experiments demonstrate state-of-the-art performance against existing methods on the Chinese SLR500 (reaching 97.1%) and Turkish AUTSL datasets (97.07% accuracy). The method's cross-lingual effectiveness highlight its potential for developing inclusive communication technologies.