EELE: Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech

TL;DR

This paper introduces EELE, a flexible method for integrating LoRA into emotional TTS models, enabling emotional capabilities without pre-embedding emotions, and optimizing insertion points and parameters for better performance.

Contribution

The paper proposes a novel, extensible LoRA integration scheme for emotional TTS that allows flexible insertion and fine-tuning, improving emotional generation without pre-embedding emotions.

Findings

LoRA can effectively learn various emotions in TTS.

Optimal insertion points vary depending on the emotion.

LoRA integration outperforms full fine-tuning in efficiency.

Abstract

In the current era of Artificial Intelligence Generated Content (AIGC), a Low-Rank Adaptation (LoRA) method has emerged. It uses a plugin-based approach to learn new knowledge with lower parameter quantities and computational costs, and it can be plugged in and out based on the specific sub-tasks, offering high flexibility. However, the current application schemes primarily incorporate LoRA into the pre-introduced conditional parts of the speech models. This fixes the position of LoRA, limiting the flexibility and scalability of its application. Therefore, we propose the Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech (EELE) method. Starting from a general neutral speech model, we do not pre-introduce emotional information but instead use the LoRA plugin to design a flexible adaptive scheme that endows the model with emotional generation capabilities. Specifically, we initially train the model using only neutral speech data. After training is complete, we insert LoRA into different modules and fine-tune the model with emotional speech data to find the optimal insertion scheme. Through experiments, we compare and test the effects of inserting LoRA at different positions within the model and assess LoRA's ability to learn various emotions, effectively proving the validity of our method. Additionally, we explore the impact of the rank size of LoRA and the difference compared to directly fine-tuning the entire model.