Self-Sufficient Framework for Continuous Sign Language Recognition

TL;DR

This paper introduces a self-sufficient framework for continuous sign language recognition that effectively extracts multi-scale features and refines pseudo-labels, achieving state-of-the-art results without additional annotations.

Contribution

It proposes novel methods (DFConv and DPLR) that enable sign language recognition using only RGB data and sequence labels, eliminating the need for complex annotations or multi-modality.

Findings

Achieves state-of-the-art performance on CSLR benchmarks

Outperforms multi-modality methods in efficiency

Comparable results with fewer annotations

Abstract

The goal of this work is to develop self-sufficient framework for Continuous Sign Language Recognition (CSLR) that addresses key issues of sign language recognition. These include the need for complex multi-scale features such as hands, face, and mouth for understanding, and absence of frame-level annotations. To this end, we propose (1) Divide and Focus Convolution (DFConv) which extracts both manual and non-manual features without the need for additional networks or annotations, and (2) Dense Pseudo-Label Refinement (DPLR) which propagates non-spiky frame-level pseudo-labels by combining the ground truth gloss sequence labels with the predicted sequence. We demonstrate that our model achieves state-of-the-art performance among RGB-based methods on large-scale CSLR benchmarks, PHOENIX-2014 and PHOENIX-2014-T, while showing comparable results with better efficiency when compared to other approaches that use multi-modality or extra annotations.