Multilingual Prosody Transfer: Comparing Supervised & Transfer Learning

TL;DR

This paper compares supervised fine-tuning and transfer learning methods for multilingual prosody transfer in speech synthesis, showing transfer learning significantly improves quality and accuracy, aiding low-resource language TTS development.

Contribution

It provides a comparative analysis of SFT and TL methods for multilingual TTS, demonstrating transfer learning's superior performance across key metrics.

Findings

Transfer learning outperforms supervised fine-tuning in MOS, RA, and MCD.

Transfer learning increases MOS by 1.53 points on average.

Transfer learning improves recognition accuracy by 37.5%.

Abstract

The field of prosody transfer in speech synthesis systems is rapidly advancing. This research is focused on evaluating learning methods for adapting pre-trained monolingual text-to-speech (TTS) models to multilingual conditions, i.e., Supervised Fine-Tuning (SFT) and Transfer Learning (TL). This comparison utilizes three distinct metrics: Mean Opinion Score (MOS), Recognition Accuracy (RA), and Mel Cepstral Distortion (MCD). Results demonstrate that, in comparison to SFT, TL leads to significantly enhanced performance, with an average MOS higher by 1.53 points, a 37.5% increase in RA, and approximately a 7.8-point improvement in MCD. These findings are instrumental in helping build TTS models for low-resource languages.