A Reference-less Quality Metric for Automatic Speech Recognition via Contrastive-Learning of a Multi-Language Model with Self-Supervision

TL;DR

This paper introduces a novel multi-language referenceless quality metric for ASR systems, leveraging contrastive learning with a pre-trained language model, which correlates better with WER and improves hypothesis ensembling.

Contribution

It presents a new self-supervised contrastive learning approach to fine-tune a multilingual language model for reference-less ASR quality assessment, outperforming existing perplexity-based metrics.

Findings

Higher correlation with WER than perplexity metrics

Reduces WER by over 7% in hypothesis ensembling

Effective across multiple languages and unseen datasets

Abstract

The common standard for quality evaluation of automatic speech recognition (ASR) systems is reference-based metrics such as the Word Error Rate (WER), computed using manual ground-truth transcriptions that are time-consuming and expensive to obtain. This work proposes a multi-language referenceless quality metric, which allows comparing the performance of different ASR models on a speech dataset without ground truth transcriptions. To estimate the quality of ASR hypotheses, a pre-trained language model (LM) is fine-tuned with contrastive learning in a self-supervised learning manner. In experiments conducted on several unseen test datasets consisting of outputs from top commercial ASR engines in various languages, the proposed referenceless metric obtains a much higher correlation with WER scores and their ranks than the perplexity metric from the state-of-art multi-lingual LM in all experiments, and also reduces WER by more than $7\%$ when used for ensembling hypotheses. The fine-tuned model and experiments are made available for the reproducibility: https://github.com/aixplain/NoRefER