Word separation in continuous sign language using isolated signs and post-processing

TL;DR

This paper introduces a two-stage model combining CNN, SVD, and LSTM with post-processing to improve word boundary detection in continuous sign language recognition, addressing a key challenge in the field.

Contribution

The paper presents a novel two-stage approach that leverages isolated sign training and post-processing to effectively segment continuous sign language videos.

Findings

Effective boundary detection in continuous signs

Model trained on isolated signs generalizes to continuous videos

Improved accuracy on public ISLR datasets

Abstract

. Continuous Sign Language Recognition (CSLR) is a long challenging task in Computer Vision due to the difficulties in detecting the explicit boundaries between the words in a sign sentence. To deal with this challenge, we propose a two-stage model. In the first stage, the predictor model, which includes a combination of CNN, SVD, and LSTM, is trained with the isolated signs. In the second stage, we apply a post-processing algorithm to the Softmax outputs obtained from the first part of the model in order to separate the isolated signs in the continuous signs. While the proposed model is trained on the isolated sign classes with similar frame numbers, it is evaluated on the continuous sign videos with a different frame length per each isolated sign class. Due to the lack of a large dataset, including both the sign sequences and the corresponding isolated signs, two public datasets in Isolated Sign Language Recognition (ISLR), RKS-PERSIANSIGN and ASLLVD, are used for evaluation. Results of the continuous sign videos confirm the efficiency of the proposed model to deal with isolated sign boundaries detection.