An Open-Source American Sign Language Fingerspell Recognition and Semantic Pose Retrieval Interface

TL;DR

This paper presents an open-source system combining CNNs and pose estimation for real-time ASL fingerspell recognition and semantic pose retrieval, aiming to facilitate sign language translation and accessibility.

Contribution

It introduces a modular, accessible interface for ASL fingerspelling recognition and pose retrieval using deep learning, advancing towards practical sign language translation tools.

Findings

System operates in real-time under diverse conditions

Achieves accurate fingerspell recognition and pose retrieval

Provides a foundation for future sign language translation applications

Abstract

This paper introduces an open-source interface for American Sign Language fingerspell recognition and semantic pose retrieval, aimed to serve as a stepping stone towards more advanced sign language translation systems. Utilizing a combination of convolutional neural networks and pose estimation models, the interface provides two modular components: a recognition module for translating ASL fingerspelling into spoken English and a production module for converting spoken English into ASL pose sequences. The system is designed to be highly accessible, user-friendly, and capable of functioning in real-time under varying environmental conditions like backgrounds, lighting, skin tones, and hand sizes. We discuss the technical details of the model architecture, application in the wild, as well as potential future enhancements for real-world consumer applications.