Fingerspelling within Sign Language Translation
Garrett Tanzer

TL;DR
This paper evaluates and improves fingerspelling recognition within sign language translation models, highlighting the effectiveness of character-level tokenization and data mixing strategies for better understanding of fingerspelling in sentences.
Contribution
It introduces a manual annotation of fingerspelling in sign language translation and assesses simple methods to enhance fingerspelling recognition in translation models.
Findings
Character-level tokenization significantly improves fingerspelling understanding.
Mixing fingerspelling data into training has mixed effects.
Enhanced fingerspelling recognition improves overall translation quality.
Abstract
Fingerspelling poses challenges for sign language processing due to its high-frequency motion and use for open-vocabulary terms. While prior work has studied fingerspelling recognition, there has been little attention to evaluating how well sign language translation models understand fingerspelling in the context of entire sentences -- and improving this capability. We manually annotate instances of fingerspelling within FLEURS-ASL and use them to evaluate the effect of two simple measures to improve fingerspelling recognition within American Sign Language to English translation: 1) use a model family (ByT5) with character- rather than subword-level tokenization, and 2) mix fingerspelling recognition data into the translation training mixture. We find that 1) substantially improves understanding of fingerspelling (and therefore translation quality overall), but the effect of 2) is mixed.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsHearing Impairment and Communication · Hand Gesture Recognition Systems · Swearing, Euphemism, Multilingualism
MethodsSoftmax · Attention Is All You Need · 7 Fastest Ways to Call American Airlines Reservations Number (USA Guide)
