PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems
Nima Sedghiyeh, Sara Sadeghi, Reza Khodadadi, Farzin Kashani, Omid Aghdaei, Somayeh Rahimi, Mohammad Sadegh Safari

TL;DR
This paper presents PSRB, a comprehensive benchmark for evaluating Persian ASR systems, highlighting performance gaps across diverse conditions and proposing a new error-weighted metric to improve evaluation accuracy.
Contribution
Introduction of PSRB, a detailed benchmark for Persian ASR, including a novel error weighting metric and analysis of model biases and error types.
Findings
ASR models perform well on standard Persian but struggle with accents and children's speech.
The new metric reduces minor error impact, improving evaluation robustness.
Performance varies significantly across linguistic and acoustic conditions.
Abstract
Although Automatic Speech Recognition (ASR) systems have become an integral part of modern technology, their evaluation remains challenging, particularly for low-resource languages such as Persian. This paper introduces Persian Speech Recognition Benchmark(PSRB), a comprehensive benchmark designed to address this gap by incorporating diverse linguistic and acoustic conditions. We evaluate ten ASR systems, including state-of-the-art commercial and open-source models, to examine performance variations and inherent biases. Additionally, we conduct an in-depth analysis of Persian ASR transcriptions, identifying key error types and proposing a novel metric that weights substitution errors. This metric enhances evaluation robustness by reducing the impact of minor and partial errors, thereby improving the precision of performance assessment. Our findings indicate that while ASR models…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFault Detection and Control Systems
