Interpretable Style Transfer for Text-to-Speech with ControlVAE and Diffusion Bridge

TL;DR

This paper introduces an interpretable style transfer system for Text-to-Speech that combines ControlVAE and diffusion models to enhance style transfer fidelity and interpretability, validated on LibriTTS.

Contribution

The paper proposes a novel TTS system integrating ControlVAE and diffusion bridge for improved style transfer and interpretability, with both one-stage and two-stage configurations.

Findings

Outperforms baseline models in style transfer quality

Effective learning of complex discrete style representations

Enhanced interpretability and reconstruction quality

Abstract

With the demand for autonomous control and personalized speech generation, the style control and transfer in Text-to-Speech (TTS) is becoming more and more important. In this paper, we propose a new TTS system that can perform style transfer with interpretability and high fidelity. Firstly, we design a TTS system that combines variational autoencoder (VAE) and diffusion refiner to get refined mel-spectrograms. Specifically, a two-stage and a one-stage system are designed respectively, to improve the audio quality and the performance of style transfer. Secondly, a diffusion bridge of quantized VAE is designed to efficiently learn complex discrete style representations and improve the performance of style transfer. To have a better ability of style transfer, we introduce ControlVAE to improve the reconstruction quality and have good interpretability simultaneously. Experiments on LibriTTS dataset demonstrate that our method is more effective than baseline models.