Pheme: Efficient and Conversational Speech Generation

TL;DR

Pheme introduces a compact, efficient, and high-quality conversational speech generation model that enables parallel, real-time synthesis using smaller training data, surpassing existing autoregressive TTS models in performance and efficiency.

Contribution

The paper presents the Pheme model series, which achieves high-quality, parallel speech synthesis with reduced data requirements and training time, addressing limitations of current autoregressive TTS systems.

Findings

Pheme models match the quality of larger autoregressive TTS models.

They enable real-time, parallel speech generation.

Training data requirements are reduced by over 10 times.

Abstract

In recent years, speech generation has seen remarkable progress, now achieving one-shot generation capability that is often virtually indistinguishable from real human voice. Integrating such advancements in speech generation with large language models might revolutionize a wide range of applications. However, certain applications, such as assistive conversational systems, require natural and conversational speech generation tools that also operate efficiently in real time. Current state-of-the-art models like VALL-E and SoundStorm, powered by hierarchical neural audio codecs, require large neural components and extensive training data to work well. In contrast, MQTTS aims to build more compact conversational TTS models while capitalizing on smaller-scale real-life conversational speech data. However, its autoregressive nature yields high inference latency and thus limits its real-time usage. In order to mitigate the current limitations of the state-of-the-art TTS models while capitalizing on their strengths, in this work we introduce the Pheme model series that 1) offers compact yet high-performing models, 2) allows for parallel speech generation of 3) natural conversational speech, and 4) it can be trained efficiently on smaller-scale conversational data, cutting data demands by more than 10x but still matching the quality of the autoregressive TTS models. We also show that through simple teacher-student distillation we can meet significant improvements in voice quality for single-speaker setups on top of pretrained Pheme checkpoints, relying solely on synthetic speech generated by much larger teacher models. Audio samples and pretrained models are available online.