Exploring Sub-skeleton Trajectories for Interpretable Recognition of Sign Language

TL;DR

This paper introduces a simple, scalable method using sub-skeleton trajectories and natural distance measures for interpretable sign language recognition, outperforming recent state-of-the-art approaches across various datasets.

Contribution

The paper proposes a novel geometric feature space called sub-skeletons and a natural distance measure for sign language recognition, demonstrating improved accuracy and interpretability.

Findings

Simple methods outperform recent state-of-the-art approaches.

Effective across different data domains and tracking technologies.

Boosted variation further improves recognition accuracy.

Abstract

Recent advances in tracking sensors and pose estimation software enable smart systems to use trajectories of skeleton joint locations for supervised learning. We study the problem of accurately recognizing sign language words, which is key to narrowing the communication gap between hard and non-hard of hearing people. Our method explores a geometric feature space that we call `sub-skeleton' aspects of movement. We assess similarity of feature space trajectories using natural, speed invariant distance measures, which enables clear and insightful nearest neighbor classification. The simplicity and scalability of our basic method allows for immediate application in different data domains with little to no parameter tuning. We demonstrate the effectiveness of our basic method, and a boosted variation, with experiments on data from different application domains and tracking technologies. Surprisingly, our simple methods improve sign recognition over recent, state-of-the-art approaches.