Isharah: A Large-Scale Multi-Scene Dataset for Continuous Sign Language Recognition

TL;DR

Isharah is a large, diverse, unconstrained multi-scene dataset for continuous sign language recognition, enabling more robust real-world CSLR and translation systems through extensive annotations and benchmarks.

Contribution

This paper introduces Isharah, the first large-scale, multi-scene CSLR dataset collected in real-world conditions with rich annotations and multiple benchmarks for sign language understanding.

Findings

Dataset contains 30,000 videos from 18 signers.

Includes benchmarks for signer-independent and unseen-sentence CSLR.

Supports development of sign language translation systems.

Abstract

Current benchmarks for sign language recognition (SLR) focus mainly on isolated SLR, while there are limited datasets for continuous SLR (CSLR), which recognizes sequences of signs in a video. Additionally, existing CSLR datasets are collected in controlled settings, which restricts their effectiveness in building robust real-world CSLR systems. To address these limitations, we present Isharah, a large multi-scene dataset for CSLR. It is the first dataset of its type and size that has been collected in an unconstrained environment using signers' smartphone cameras. This setup resulted in high variations of recording settings, camera distances, angles, and resolutions. This variation helps with developing sign language understanding models capable of handling the variability and complexity of real-world scenarios. The dataset consists of 30,000 video clips performed by 18 deaf and professional signers. Additionally, the dataset is linguistically rich as it provides a gloss-level annotation for all dataset's videos, making it useful for developing CSLR and sign language translation (SLT) systems. This paper also introduces multiple sign language understanding benchmarks, including signer-independent and unseen-sentence CSLR, along with gloss-based and gloss-free SLT. The Isharah dataset is available on https://snalyami.github.io/Isharah_CSLR/.