RepeaTTS: Towards Feature Discovery through Repeated Fine-Tuning

TL;DR

This paper introduces RepeaTTS, a novel fine-tuning approach that leverages uncontrollable variance to discover latent speech features, enhancing controllability in prompt-based text-to-speech models.

Contribution

It proposes a new fine-tuning regime using principal component analysis to identify and incorporate latent features, improving control over speech synthesis.

Findings

Latent features improve speech controllability.

Method works on expressive Icelandic speech data.

Both continuous and discrete features are effective.

Abstract

A Prompt-based Text-To-Speech model allows a user to control different aspects of speech, such as speaking rate and perceived gender, through natural language instruction. Although user-friendly, such approaches are on one hand constrained: control is limited to acoustic features exposed to the model during training, and too flexible on the other: the same inputs yields uncontrollable variation that are reflected in the corpus statistics. We investigate a novel fine-tuning regime to address both of these issues at the same time by exploiting the uncontrollable variance of the model. Through principal component analysis of thousands of synthesised samples, we determine latent features that account for the highest proportion of the output variance and incorporate them as new labels for secondary fine-tuning. We evaluate the proposed methods on two models trained on an expressive Icelandic speech corpus, one with emotional disclosure and one without. In the case of the model without emotional disclosure, the method yields both continuous and discrete features that improve overall controllability of the model.