Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis

TL;DR

This paper introduces HE-Vector, a hierarchical method for emotionally expressive dialectal TTS that independently models dialect and emotion, enabling controllable synthesis without joint labels, improving naturalness and expressiveness.

Contribution

The paper proposes a novel hierarchical vector approach for zero-shot emotionally expressive dialect TTS, addressing data scarcity and enabling independent style control.

Findings

HE-Vector outperforms baseline in dialect synthesis quality.

Promising zero-shot emotional dialect synthesis results.

Independent style modeling enhances controllability.

Abstract

Recent advances in text-to-speech (TTS) have yielded remarkable improvements in naturalness and intelligibility. Building on these achievements, research has increasingly shifted toward enhancing the expressiveness of generated speech, such as dialectal and emotional TTS. However, cross-style synthesis combining both dialect and emotion remains challenging and largely unexplored, mainly due to the scarcity of dialectal data with emotional labels. To address this, we propose Hierarchical Expressive Vector (HE-Vector), a two-stage method for Emotional Dialectal TTS. In the first stage, we construct different task vectors to model dialectal and emotional styles independently, and then enhance single-style synthesis by adjusting their weights, a method we refer to as Expressive Vector (E-Vector). For the second stage, we hierarchically integrate these vectors to achieve controllable emotionally expressive dialect synthesis without requiring jointly labeled data, corresponding to Hierarchical Expressive Vector (HE-Vector). Experimental results demonstrate that HE-Vectors achieve superior performance in dialect synthesis, and promising results in synthesizing emotionally expressive dialectal speech in a zero-shot setting.