Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning

TL;DR

This paper introduces Voicebox Adapter, an efficient fine-tuning method that enhances controllability in speech generation models by integrating fine-grained conditions with minimal resource use.

Contribution

It presents a novel cross-attention based adapter for pre-trained speech models and identifies LoRA with bias-tuning as the most effective fine-tuning approach.

Findings

LoRA with bias-tuning achieves optimal controllability and speech quality.

Voicebox Adapter improves resource efficiency in fine-grained speech generation.

The method demonstrates robustness across various datasets.

Abstract

As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a cross-attention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups.