RedPen: Region- and Reason-Annotated Dataset of Unnatural Speech

TL;DR

RedPen is a new speech dataset with detailed human annotations of unnatural speech regions and their error types, aiming to improve interpretability in speech synthesis evaluation.

Contribution

The paper introduces RedPen, a novel dataset with human-annotated unnatural speech regions and reasons, enhancing analysis of speech synthesis errors beyond traditional scores.

Findings

RedPen annotations provide better explanations for unnatural speech regions.

Different speech synthesis models exhibit distinct error types.

Dataset enables more interpretable evaluation of speech synthesis quality.

Abstract

Even with recent advances in speech synthesis models, the evaluation of such models is based purely on human judgement as a single naturalness score, such as the Mean Opinion Score (MOS). The score-based metric does not give any further information about which parts of speech are unnatural or why human judges believe they are unnatural. We present a novel speech dataset, RedPen, with human annotations on unnatural speech regions and their corresponding reasons. RedPen consists of 180 synthesized speeches with unnatural regions annotated by crowd workers; These regions are then reasoned and categorized by error types, such as voice trembling and background noise. We find that our dataset shows a better explanation for unnatural speech regions than the model-driven unnaturalness prediction. Our analysis also shows that each model includes different types of error types. Summing up, our dataset successfully shows the possibility that various error regions and types lie under the single naturalness score. We believe that our dataset will shed light on the evaluation and development of more interpretable speech models in the future. Our dataset will be publicly available upon acceptance.