Accent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data

TL;DR

This paper introduces Accent Vector, a novel method for controllable accent manipulation in multilingual TTS systems that does not require accented data, enabling fine-grained and cross-lingual accent control.

Contribution

The paper presents a new accent representation technique that allows accent control in TTS without needing accented training data, using fine-tuning and task vectors.

Findings

Effective fine-grained accent control demonstrated

Cross-lingual accent manipulation achieved

Objective and human evaluations confirm success

Abstract

Accent is an integral part of society, reflecting multiculturalism and shaping how individuals express identity. The majority of English speakers are non-native (L2) speakers, yet current Text-To-Speech (TTS) systems primarily model American-accented English due limited accented data. We propose \textit{Accent Vector}, a controllable representation that enables accent manipulation in multilingual TTS without requiring accented training data. \textit{Accent Vector} is derived by fine-tuning a TTS system on native speech of a different language (i.e. non-English) and computing task vectors capturing accent characteristics (i.e. in English). By scaling and interpolating the vector, we achieve fine-grained control over accent strength and generate mixed-accent speech. In addition, it generalizes beyond English, enabling accent control across multiple languages. Objective and human evaluations confirm the effectiveness of Accent Vector for fine-grained and compositional accent control.