Open Universal Arabic ASR Leaderboard

TL;DR

This paper introduces a comprehensive benchmark for open-source Arabic speech recognition models across multiple dialects, evaluating their performance, robustness, and efficiency to advance the development of universal Arabic ASR systems.

Contribution

It presents the Open Universal Arabic ASR Leaderboard, establishing a standardized evaluation framework for multi-dialect Arabic ASR models and providing detailed analysis of their capabilities.

Findings

Benchmark covers diverse multi-dialect datasets

Analysis includes robustness, speaker adaptation, efficiency, and memory use

Provides a reference for model performance and generalization

Abstract

In recent years, the enhanced capabilities of ASR models and the emergence of multi-dialect datasets have increasingly pushed Arabic ASR model development toward an all-dialect-in-one direction. This trend highlights the need for benchmarking studies that evaluate model performance on multiple dialects, providing the community with insights into models' generalization capabilities. In this paper, we introduce Open Universal Arabic ASR Leaderboard, a continuous benchmark project for open-source general Arabic ASR models across various multi-dialect datasets. We also provide a comprehensive analysis of the model's robustness, speaker adaptation, inference efficiency, and memory consumption. This work aims to offer the Arabic ASR community a reference for models' general performance and also establish a common evaluation framework for multi-dialectal Arabic ASR models.