Federated Learning with Dynamic Transformer for Text to Speech

TL;DR

This paper introduces a federated dynamic transformer for text-to-speech that improves training stability, reduces communication costs, and approaches centralized model performance, addressing privacy and efficiency challenges in federated learning.

Contribution

It proposes a novel federated dynamic transformer model that enhances convergence speed, stability, and communication efficiency for federated TTS systems.

Findings

Achieves near-centralized performance with more clients.

Faster and more stable training convergence.

Significantly reduces communication overhead.

Abstract

Text to speech (TTS) is a crucial task for user interaction, but TTS model training relies on a sizable set of high-quality original datasets. Due to privacy and security issues, the original datasets are usually unavailable directly. Recently, federated learning proposes a popular distributed machine learning paradigm with an enhanced privacy protection mechanism. It offers a practical and secure framework for data owners to collaborate with others, thus obtaining a better global model trained on the larger dataset. However, due to the high complexity of transformer models, the convergence process becomes slow and unstable in the federated learning setting. Besides, the transformer model trained in federated learning is costly communication and limited computational speed on clients, impeding its popularity. To deal with these challenges, we propose the federated dynamic transformer. On the one hand, the performance is greatly improved comparing with the federated transformer, approaching centralize-trained Transformer-TTS when increasing clients number. On the other hand, it achieves faster and more stable convergence in the training phase and significantly reduces communication time. Experiments on the LJSpeech dataset also strongly prove our method's advantage.