Swiss German Speech to Text system evaluation

TL;DR

This paper evaluates four commercial Swiss German speech-to-text systems against a custom model across two datasets, revealing the custom model's superior performance through detailed error analysis and BLEU score comparisons.

Contribution

The study provides a comprehensive comparison of commercial and custom Swiss German STT systems, including training details and error analysis, highlighting the custom model's improved accuracy.

Findings

The custom FHNW model outperforms commercial systems in BLEU scores.

Error analysis reveals specific strengths and weaknesses of each system.

The custom model achieves BLEU scores of 0.607 and 0.722 on two datasets.

Abstract

We present an in-depth evaluation of four commercially available Speech-to-Text (STT) systems for Swiss German. The systems are anonymized and referred to as system a-d in this report. We compare the four systems to our STT model, referred to as FHNW from hereon after, and provide details on how we trained our model. To evaluate the models, we use two STT datasets from different domains. The Swiss Parliament Corpus (SPC) test set and a private dataset in the news domain with an even distribution across seven dialect regions. We provide a detailed error analysis to detect the three systems' strengths and weaknesses. This analysis is limited by the characteristics of the two test sets. Our model scored the highest bilingual evaluation understudy (BLEU) on both datasets. On the SPC test set, we obtain a BLEU score of 0.607, whereas the best commercial system reaches a BLEU score of 0.509. On our private test set, we obtain a BLEU score of 0.722 and the best commercial system a BLEU score of 0.568.