Phir Hera Fairy: An English Fairytaler is a Strong Faker of Fluent Speech in Low-Resource Indian Languages

TL;DR

This paper evaluates how fine-tuning an English TTS model on Indian languages enhances multilingual fluency, voice cloning, and zero-resource synthesis, demonstrating near-human performance in low-resource settings.

Contribution

It introduces IN-F5, a fine-tuned TTS model that achieves high-quality multilingual synthesis and zero-resource language generation with minimal data, advancing low-resource TTS capabilities.

Findings

Fine-tuning on Indian data yields near-human polyglot TTS.

English pretraining benefits low-resource language synthesis.

IN-F5 can synthesize unseen languages like Bhojpuri and Tulu.

Abstract

What happens when an English Fairytaler is fine-tuned on Indian languages? We evaluate how the English F5-TTS model adapts to 11 Indian languages, measuring polyglot fluency, voice-cloning, style-cloning, and code-mixing. We compare: (i) training from scratch, (ii) fine-tuning English F5 on Indian data, and (iii) fine-tuning on both Indian and English data to prevent forgetting. Fine-tuning with only Indian data proves most effective and the resultant IN-F5 is a near-human polyglot; that enables speakers of one language (e.g., Odia) to fluently speak in another (e.g., Hindi). Our results show English pretraining aids low-resource TTS in reaching human parity. To aid progress in other low-resource languages, we study data-constrained setups and arrive at a compute optimal strategy. Finally, we show IN-F5 can synthesize unseen languages like Bhojpuri and Tulu using a human-in-the-loop approach for zero-resource TTS via synthetic data generation.