Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakers

TL;DR

This paper introduces an adapter-based method for adapting multi-speaker TTS models to new speakers efficiently, avoiding extensive fine-tuning and preserving original speaker quality.

Contribution

It proposes a parameter-efficient adapter module approach that enables quick adaptation to new speakers without degrading existing speaker quality.

Findings

Effective adaptation demonstrated on LibriTTS, HiFi-TTS, and VCTK datasets.

Achieved high-quality speech synthesis for new speakers with minimal parameter updates.

Preserved original speaker quality while adding new speakers.

Abstract

Fine-tuning is a popular method for adapting text-to-speech (TTS) models to new speakers. However this approach has some challenges. Usually fine-tuning requires several hours of high quality speech per speaker. There is also that fine-tuning will negatively affect the quality of speech synthesis for previously learnt speakers. In this paper we propose an alternative approach for TTS adaptation based on using parameter-efficient adapter modules. In the proposed approach, a few small adapter modules are added to the original network. The original weights are frozen, and only the adapters are fine-tuned on speech for new speaker. The parameter-efficient fine-tuning approach will produce a new model with high level of parameter sharing with original model. Our experiments on LibriTTS, HiFi-TTS and VCTK datasets validate the effectiveness of adapter-based method through objective and subjective metrics.