ParrotTTS: Text-to-Speech synthesis by exploiting self-supervised representations

TL;DR

ParrotTTS is a modular text-to-speech model that uses self-supervised speech representations to enable effective multi-speaker, multilingual, and cross-lingual voice synthesis with limited data.

Contribution

It introduces a novel TTS approach leveraging disentangled self-supervised representations, enabling low-resource language adaptation and cross-language voice transfer without parallel data.

Findings

Outperforms state-of-the-art multi-lingual TTS models with less paired data

Effective in low-resource and unseen language scenarios

Capable of voice transfer across languages while maintaining speaker identity

Abstract

We present ParrotTTS, a modularized text-to-speech synthesis model leveraging disentangled self-supervised speech representations. It can train a multi-speaker variant effectively using transcripts from a single speaker. ParrotTTS adapts to a new language in low resource setup and generalizes to languages not seen while training the self-supervised backbone. Moreover, without training on bilingual or parallel examples, ParrotTTS can transfer voices across languages while preserving the speaker specific characteristics, e.g., synthesizing fluent Hindi speech using a French speaker's voice and accent. We present extensive results in monolingual and multi-lingual scenarios. ParrotTTS outperforms state-of-the-art multi-lingual TTS models using only a fraction of paired data as latter.