TTSDS -- Text-to-Speech Distribution Score

TL;DR

The paper introduces TTSDS, a comprehensive evaluation score for synthetic speech that assesses multiple quality factors and correlates well with human judgments across various TTS systems and time periods.

Contribution

It proposes a new multi-factor evaluation method for TTS quality that considers prosody, speaker identity, and intelligibility, improving the assessment of synthetic speech.

Findings

TTSDS score strongly correlates with human evaluations.

Benchmarking 35 TTS systems from 2008 to 2024 demonstrates the score's effectiveness.

The method provides a nuanced assessment of synthetic speech quality.

Abstract

Many recently published Text-to-Speech (TTS) systems produce audio close to real speech. However, TTS evaluation needs to be revisited to make sense of the results obtained with the new architectures, approaches and datasets. We propose evaluating the quality of synthetic speech as a combination of multiple factors such as prosody, speaker identity, and intelligibility. Our approach assesses how well synthetic speech mirrors real speech by obtaining correlates of each factor and measuring their distance from both real speech datasets and noise datasets. We benchmark 35 TTS systems developed between 2008 and 2024 and show that our score computed as an unweighted average of factors strongly correlates with the human evaluations from each time period.