On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis

TL;DR

This paper investigates the effects of pruning on end-to-end text-to-speech models, revealing that significant pruning can maintain or even improve speech naturalness, intelligibility, and prosody, supported by extensive subjective and objective evaluations.

Contribution

It demonstrates that end-to-end TTS models are highly prunable without loss of quality, and explores methods like finetuning data, TTS-augmentation, and knowledge distillation to optimize pruning strategies.

Findings

Pruned TTS models can match or surpass original models in naturalness.

Pruning does not necessarily degrade speech quality, and can sometimes enhance it.

The study provides a large dataset of pruned models and evaluation results.

Abstract

Are end-to-end text-to-speech (TTS) models over-parametrized? To what extent can these models be pruned, and what happens to their synthesis capabilities? This work serves as a starting point to explore pruning both spectrogram prediction networks and vocoders. We thoroughly investigate the tradeoffs between sparsity and its subsequent effects on synthetic speech. Additionally, we explored several aspects of TTS pruning: amount of finetuning data versus sparsity, TTS-Augmentation to utilize unspoken text, and combining knowledge distillation and pruning. Our findings suggest that not only are end-to-end TTS models highly prunable, but also, perhaps surprisingly, pruned TTS models can produce synthetic speech with equal or higher naturalness and intelligibility, with similar prosody. All of our experiments are conducted on publicly available models, and findings in this work are backed by large-scale subjective tests and objective measures. Code and 200 pruned models are made available to facilitate future research on efficiency in TTS.