A Transformer Model for Boundary Detection in Continuous Sign Language

TL;DR

This paper introduces a Transformer-based model for boundary detection in continuous sign language, improving accuracy without handcrafted features and applicable to both isolated and continuous recognition tasks.

Contribution

The paper presents a novel Transformer approach that enhances boundary detection in CSLR without relying on handcrafted features, applicable to both ISLR and CSLR.

Findings

Promising results on two sign language datasets

Effective boundary detection in continuous sign videos

Elimination of handcrafted feature reliance

Abstract

Sign Language Recognition (SLR) has garnered significant attention from researchers in recent years, particularly the intricate domain of Continuous Sign Language Recognition (CSLR), which presents heightened complexity compared to Isolated Sign Language Recognition (ISLR). One of the prominent challenges in CSLR pertains to accurately detecting the boundaries of isolated signs within a continuous video stream. Additionally, the reliance on handcrafted features in existing models poses a challenge to achieving optimal accuracy. To surmount these challenges, we propose a novel approach utilizing a Transformer-based model. Unlike traditional models, our approach focuses on enhancing accuracy while eliminating the need for handcrafted features. The Transformer model is employed for both ISLR and CSLR. The training process involves using isolated sign videos, where hand keypoint features extracted from the input video are enriched using the Transformer model. Subsequently, these enriched features are forwarded to the final classification layer. The trained model, coupled with a post-processing method, is then applied to detect isolated sign boundaries within continuous sign videos. The evaluation of our model is conducted on two distinct datasets, including both continuous signs and their corresponding isolated signs, demonstrates promising results.