Convert and Speak: Zero-shot Accent Conversion with Minimum Supervision

TL;DR

This paper introduces a two-stage zero-shot accent conversion framework that uses minimal supervision and semantic tokens, enabling high-quality accent conversion with very limited parallel data.

Contribution

The proposed 'convert-and-speak' framework decouples accent conversion and speech synthesis, reducing data requirements and leveraging language pre-training for effective zero-shot accent conversion.

Findings

Achieves state-of-the-art accent similarity and speech quality.

Operates effectively with only 15 minutes of weakly parallel data.

Demonstrates high adaptability across diverse accents.

Abstract

Low resource of parallel data is the key challenge of accent conversion(AC) problem in which both the pronunciation units and prosody pattern need to be converted. We propose a two-stage generative framework "convert-and-speak" in which the conversion is only operated on the semantic token level and the speech is synthesized conditioned on the converted semantic token with a speech generative model in target accent domain. The decoupling design enables the "speaking" module to use massive amount of target accent speech and relieves the parallel data required for the "conversion" module. Conversion with the bridge of semantic token also relieves the requirement for the data with text transcriptions and unlocks the usage of language pre-training technology to further efficiently reduce the need of parallel accent speech data. To reduce the complexity and latency of "speaking", a single-stage AR generative model is designed to achieve good quality as well as lower computation cost. Experiments on Indian-English to general American-English conversion show that the proposed framework achieves state-of-the-art performance in accent similarity, speech quality, and speaker maintenance with only 15 minutes of weakly parallel data which is not constrained to the same speaker. Extensive experimentation with diverse accent types suggests that this framework possesses a high degree of adaptability, making it readily scalable to accommodate other accents with low-resource data. Audio samples are available at https://www.microsoft.com/en-us/research/project/convert-and-speak-zero-shot-accent-conversion-with-minimumsupervision/.