LoRP-TTS: Low-Rank Personalized Text-To-Speech

TL;DR

LoRP-TTS leverages Low-Rank Adaptation to improve zero-shot personalized speech synthesis, enabling high-quality imitation from minimal and noisy spontaneous speech samples, thus advancing diversity in speech corpora.

Contribution

This work introduces the use of Low-Rank Adaptation in TTS to effectively utilize single noisy recordings for speaker adaptation, enhancing realism and diversity.

Findings

Speaker similarity improved by up to 30 percentage points

Effective use of noisy, spontaneous speech samples as prompts

Advances in creating diverse speech datasets

Abstract

Speech synthesis models convert written text into natural-sounding audio. While earlier models were limited to a single speaker, recent advancements have led to the development of zero-shot systems that generate realistic speech from a wide range of speakers using their voices as additional prompts. However, they still struggle with imitating non-studio-quality samples that differ significantly from the training datasets. In this work, we demonstrate that utilizing Low-Rank Adaptation (LoRA) allows us to successfully use even single recordings of spontaneous speech in noisy environments as prompts. This approach enhances speaker similarity by up to $30pp$ while preserving content and naturalness. It represents a significant step toward creating truly diverse speech corpora, that is crucial in all speech-related tasks.