On the Use of Self-Supervised Speech Representations in Spontaneous Speech Synthesis

TL;DR

This paper systematically compares six self-supervised speech representations across multiple layers for spontaneous speech synthesis and MOS prediction, providing insights into their optimal usage and generalizability.

Contribution

It extends the comparison of SSL models for spontaneous TTS to multiple models and layers, and evaluates SSL-based MOS prediction on spontaneous speech, offering comprehensive insights.

Findings

Certain SSL models and layers outperform others in spontaneous TTS.

SSL-based MOS prediction correlates well with human judgments in spontaneous speech.

Results are consistent across different spontaneous speech corpora.

Abstract

Self-supervised learning (SSL) speech representations learned from large amounts of diverse, mixed-quality speech data without transcriptions are gaining ground in many speech technology applications. Prior work has shown that SSL is an effective intermediate representation in two-stage text-to-speech (TTS) for both read and spontaneous speech. However, it is still not clear which SSL and which layer from each SSL model is most suited for spontaneous TTS. We address this shortcoming by extending the scope of comparison for SSL in spontaneous TTS to 6 different SSLs and 3 layers within each SSL. Furthermore, SSL has also shown potential in predicting the mean opinion scores (MOS) of synthesized speech, but this has only been done in read-speech MOS prediction. We extend an SSL-based MOS prediction framework previously developed for scoring read speech synthesis and evaluate its performance on synthesized spontaneous speech. All experiments are conducted twice on two different spontaneous corpora in order to find generalizable trends. Overall, we present comprehensive experimental results on the use of SSL in spontaneous TTS and MOS prediction to further quantify and understand how SSL can be used in spontaneous TTS. Audios samples: https://www.speech.kth.se/tts-demos/sp_ssl_tts