3D-LEX v1.0: 3D Lexicons for American Sign Language and Sign Language of the Netherlands

TL;DR

This paper introduces 3D-LEX v1.0, a comprehensive 3D sign language dataset with a semi-automatic annotation method, enabling improved sign recognition and detailed linguistic analysis for American Sign Language and Sign Language of the Netherlands.

Contribution

The work presents a new 3D sign language dataset with semi-automatic phonetic annotation, integrating multiple motion capture techniques for enhanced sign language analysis.

Findings

Handshape labels improve sign recognition accuracy by 5%.

The dataset includes 2,000 signs from two sign languages.

Motion capture enables detailed 3D sign analysis.

Abstract

In this work, we present an efficient approach for capturing sign language in 3D, introduce the 3D-LEX v1.0 dataset, and detail a method for semi-automatic annotation of phonetic properties. Our procedure integrates three motion capture techniques encompassing high-resolution 3D poses, 3D handshapes, and depth-aware facial features, and attains an average sampling rate of one sign every 10 seconds. This includes the time for presenting a sign example, performing and recording the sign, and archiving the capture. The 3D-LEX dataset includes 1,000 signs from American Sign Language and an additional 1,000 signs from the Sign Language of the Netherlands. We showcase the dataset utility by presenting a simple method for generating handshape annotations directly from 3D-LEX. We produce handshape labels for 1,000 signs from American Sign Language and evaluate the labels in a sign recognition task. The labels enhance gloss recognition accuracy by 5% over using no handshape annotations, and by 1% over expert annotations. Our motion capture data supports in-depth analysis of sign features and facilitates the generation of 2D projections from any viewpoint. The 3D-LEX collection has been aligned with existing sign language benchmarks and linguistic resources, to support studies in 3D-aware sign language processing.