Exploiting the Logits: Joint Sign Language Recognition and Spell-Correction

TL;DR

This paper presents a joint approach for sign language recognition and spell-correction using deep learning, demonstrating improved text readability despite limited training data and low character accuracy.

Contribution

It introduces a CNN-based spell-correction method that leverages logits and softmax outputs, enhancing generalization in scarce data scenarios.

Findings

Achieved 75% character accuracy on German sign language videos.

Logit-based training improves generalization over softmax-only methods.

The approach yields readable text despite low character accuracy.

Abstract

Machine learning techniques have excelled in the automatic semantic analysis of images, reaching human-level performances on challenging benchmarks. Yet, the semantic analysis of videos remains challenging due to the significantly higher dimensionality of the input data, respectively, the significantly higher need for annotated training examples. By studying the automatic recognition of German sign language videos, we demonstrate that on the relatively scarce training data of 2.800 videos, modern deep learning architectures for video analysis (such as ResNeXt) along with transfer learning on large gesture recognition tasks, can achieve about 75% character accuracy. Considering that this leaves us with a probability of under 25% that a 5 letter word is spelled correctly, spell-correction systems are crucial for producing readable outputs. The contribution of this paper is to propose a convolutional neural network for spell-correction that expects the softmax outputs of the character recognition network (instead of a misspelled word) as an input. We demonstrate that purely learning on softmax inputs in combination with scarce training data yields overfitting as the network learns the inputs by heart. In contrast, training the network on several variants of the logits of the classification output i.e. scaling by a constant factor, adding of random noise, mixing of softmax and hardmax inputs or purely training on hardmax inputs, leads to better generalization while benefitting from the significant information hidden in these outputs (that have 98% top-5 accuracy), yielding a readable text despite the comparably low character accuracy.