FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation

TL;DR

This paper introduces FLEURS-ASL, a new benchmark for evaluating machine translation and understanding of American Sign Language, integrating sign language video data into multilingual evaluation frameworks.

Contribution

It extends existing multilingual benchmarks to include ASL video translation, providing a unified evaluation platform and baseline models for ASL translation tasks.

Findings

Multimodal models show limited understanding of ASL.

FLEURS-ASL enables evaluation of ASL translation across many languages.

Baseline models perform comparably to phrase-level methods.

Abstract

Sign language translation has historically been peripheral to mainstream machine translation research. In order to help converge the fields, we introduce FLEURS-ASL, an extension of the multiway parallel benchmarks FLORES (for text) and FLEURS (for speech) to support their first sign language (as video), American Sign Language, translated by 5 Certified Deaf Interpreters. FLEURS-ASL can be used to evaluate a variety of tasks -- primarily sentence- and discourse-level translation -- between ASL and 200 other languages as text, or 102 languages as speech. We provide baselines for tasks from ASL to English text using a unified modeling approach that incorporates timestamp tokens and previous text tokens in a 34-second context window, trained on random video clips from YouTube-ASL. This model meets or exceeds the performance of phrase-level baselines while supporting a multitude of new tasks. We also use FLEURS-ASL to show that multimodal frontier models have virtually no understanding of ASL, underscoring the importance of including sign languages in standard evaluation suites.