Event Stream-based Sign Language Translation: A High-Definition Benchmark Dataset and A Novel Baseline

TL;DR

This paper introduces a high-definition event-based sign language dataset and a novel translation framework, addressing limitations of traditional video-based methods and advancing AI-assisted sign language translation.

Contribution

It presents a new event-based sign language dataset, Event-CSL, and a novel translation model, EvSLT, improving robustness and performance in sign language translation tasks.

Findings

Event-CSL dataset contains 14,827 videos and covers diverse conditions.

EvSLT outperforms existing methods on multiple datasets.

The approach enhances translation accuracy and robustness.

Abstract

Sign Language Translation (SLT) is a core task in the field of AI-assisted disability. Traditional SLT methods are typically based on visible light videos, which are easily affected by factors such as lighting variations, rapid hand movements, and privacy concerns. This paper proposes the use of bio-inspired event cameras to alleviate the aforementioned issues. Specifically, we introduce a new high-definition event-based sign language dataset, termed Event-CSL, which effectively addresses the data scarcity in this research area. The dataset comprises 14,827 videos, 14,821 glosses, and 2,544 Chinese words in the text vocabulary. These samples are collected across diverse indoor and outdoor scenes, covering multiple viewpoints, lighting conditions, and camera motions. We have also benchmarked existing mainstream SLT methods on this dataset to facilitate fair comparisons in future research.Furthermore, we propose a novel event-based sign language translation framework, termed EvSLT. The framework first segments continuous video features into clips and employs a Mamba-based memory aggregation module to compress and aggregate spatial detail features at the clip level. Subsequently, these spatial features, along with temporal representations obtained from temporal convolution, are then fused by a graph-guided spatiotemporal fusion module. Extensive experiments on Event-CSL, as well as other publicly available datasets, demonstrate the superior performance of our method. The dataset and source code will be released on https://github.com/Event-AHU/OpenESL