A Signer-Invariant Conformer and Multi-Scale Fusion Transformer for Continuous Sign Language Recognition

TL;DR

This paper introduces a dual-architecture framework with a Signer-Invariant Conformer and Multi-Scale Fusion Transformer to improve continuous sign language recognition, addressing signer variability and unseen sentence structures, achieving state-of-the-art results.

Contribution

The paper presents novel signer-invariant and multi-scale fusion models specifically designed for CSLR, significantly enhancing recognition accuracy and generalization to new signers and sentences.

Findings

Achieved 13.07% WER on SI challenge, outperforming previous methods.

Scored 47.78% WER on US task, surpassing prior work.

Placed 2nd in SignEval 2025 US task, 4th in SI task.

Abstract

Continuous Sign Language Recognition (CSLR) faces multiple challenges, including significant inter-signer variability and poor generalization to novel sentence structures. Traditional solutions frequently fail to handle these issues efficiently. For overcoming these constraints, we propose a dual-architecture framework. For the Signer-Independent (SI) challenge, we propose a Signer-Invariant Conformer that combines convolutions with multi-head self-attention to learn robust, signer-agnostic representations from pose-based skeletal keypoints. For the Unseen-Sentences (US) task, we designed a Multi-Scale Fusion Transformer with a novel dual-path temporal encoder that captures both fine-grained posture dynamics, enabling the model's ability to comprehend novel grammatical compositions. Experiments on the challenging Isharah-1000 dataset establish a new standard for both CSLR benchmarks. The proposed conformer architecture achieves a Word Error Rate (WER) of 13.07% on the SI challenge, a reduction of 13.53% from the state-of-the-art. On the US task, the transformer model scores a WER of 47.78%, surpassing previous work. In the SignEval 2025 CSLR challenge, our team placed 2nd in the US task and 4th in the SI task, demonstrating the performance of these models. The findings validate our key hypothesis: that developing task-specific networks designed for the particular challenges of CSLR leads to considerable performance improvements and establishes a new baseline for further research. The source code is available at: https://github.com/rezwanh001/MSLR-Pose86K-CSLR-Isharah.