Rhythm Modeling for Voice Conversion

TL;DR

This paper introduces Urhythmic, an unsupervised method for modeling and converting speech rhythm in voice conversion, improving prosody without needing parallel data or transcriptions.

Contribution

The paper presents a novel unsupervised approach to rhythm modeling in voice conversion that estimates and matches speaking rate and segment durations without supervision.

Findings

Urhythmic outperforms existing unsupervised methods in quality and prosody.

The method effectively models rhythm using self-supervised representations.

It does not require parallel data or text transcriptions.

Abstract

Voice conversion aims to transform source speech into a different target voice. However, typical voice conversion systems do not account for rhythm, which is an important factor in the perception of speaker identity. To bridge this gap, we introduce Urhythmic-an unsupervised method for rhythm conversion that does not require parallel data or text transcriptions. Using self-supervised representations, we first divide source audio into segments approximating sonorants, obstruents, and silences. Then we model rhythm by estimating speaking rate or the duration distribution of each segment type. Finally, we match the target speaking rate or rhythm by time-stretching the speech segments. Experiments show that Urhythmic outperforms existing unsupervised methods in terms of quality and prosody. Code and checkpoints: https://github.com/bshall/urhythmic. Audio demo page: https://ubisoft-laforge.github.io/speech/urhythmic.