Analytic Study of Text-Free Speech Synthesis for Raw Audio using a Self-Supervised Learning Model

TL;DR

This paper investigates text-free speech synthesis using self-supervised learning models, comparing discrete SSL representations with text, and finds that each preserves different aspects of speech quality.

Contribution

It introduces a method for speech synthesis using SSL-derived discrete symbols and analyzes their effectiveness compared to traditional text representations.

Findings

SSL representations better preserve acoustic content

Text representations better preserve semantic information

Discrete symbols outperform in prosody and intonation preservation

Abstract

We examine the text-free speech representations of raw audio obtained from a self-supervised learning (SSL) model by analyzing the synthesized speech using the SSL representations instead of conventional text representations. Since raw audio does not have paired speech representations as transcribed texts do, obtaining speech representations from unpaired speech is crucial for augmenting available datasets for speech synthesis. Specifically, the proposed speech synthesis is conducted using discrete symbol representations from the SSL model in comparison with text representations, and analytical examinations of the synthesized speech have been carried out. The results empirically show that using text representations is advantageous for preserving semantic information, while using discrete symbol representations is superior for preserving acoustic content, including prosodic and intonational information.