Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech

TL;DR

This paper proposes a novel evaluation method for synthetic speech by training speech recognition models on synthetic data and testing on real speech, providing a broader quality assessment beyond traditional intelligibility metrics.

Contribution

It introduces a new evaluation metric based on training ASR models on synthetic speech and testing on real speech, correlating well with human judgments.

Findings

The proposed metric correlates strongly with MOS scores.

It outperforms existing automatic metrics like SpeechLMScore and MOSNet.

The method is validated on three recent TTS systems.

Abstract

Modern speech synthesis systems have improved significantly, with synthetic speech being indistinguishable from real speech. However, efficient and holistic evaluation of synthetic speech still remains a significant challenge. Human evaluation using Mean Opinion Score (MOS) is ideal, but inefficient due to high costs. Therefore, researchers have developed auxiliary automatic metrics like Word Error Rate (WER) to measure intelligibility. Prior works focus on evaluating synthetic speech based on pre-trained speech recognition models, however, this can be limiting since this approach primarily measures speech intelligibility. In this paper, we propose an evaluation technique involving the training of an ASR model on synthetic speech and assessing its performance on real speech. Our main assumption is that by training the ASR model on the synthetic speech, the WER on real speech reflects the similarity between distributions, a broader assessment of synthetic speech quality beyond intelligibility. Our proposed metric demonstrates a strong correlation with both MOS naturalness and MOS intelligibility when compared to SpeechLMScore and MOSNet on three recent Text-to-Speech (TTS) systems: MQTTS, StyleTTS, and YourTTS.