signwriting-evaluation: Effective Sign Language Evaluation via SignWriting

TL;DR

This paper develops and evaluates specialized metrics for assessing SignWriting, a system for transcribing sign languages, to improve the development of sign language translation models.

Contribution

It introduces tailored evaluation metrics for SignWriting, including adaptations of existing metrics and a novel symbol distance measure, addressing unique challenges in sign language evaluation.

Findings

Metrics reveal strengths and limitations in SignWriting evaluation

Qualitative analysis demonstrates metric effectiveness

Provides tools to advance sign language processing

Abstract

The lack of automatic evaluation metrics tailored for SignWriting presents a significant obstacle in developing effective transcription and translation models for signed languages. This paper introduces a comprehensive suite of evaluation metrics specifically designed for SignWriting, including adaptations of standard metrics such as \texttt{BLEU} and \texttt{chrF}, the application of \texttt{CLIPScore} to SignWriting images, and a novel symbol distance metric unique to our approach. We address the distinct challenges of evaluating single signs versus continuous signing and provide qualitative demonstrations of metric efficacy through score distribution analyses and nearest-neighbor searches within the SignBank corpus. Our findings reveal the strengths and limitations of each metric, offering valuable insights for future advancements using SignWriting. This work contributes essential tools for evaluating SignWriting models, facilitating progress in the field of sign language processing. Our code is available at \url{https://github.com/sign-language-processing/signwriting-evaluation}.