CycleFlow: Leveraging Cycle Consistency in Flow Matching for Speaker Style Adaptation

TL;DR

CycleFlow introduces a cycle consistency-based flow matching method for non-parallel voice conversion, significantly enhancing speaker similarity and speech quality by addressing pitch and style adaptation challenges.

Contribution

The paper proposes CycleFlow, a novel cycle consistency approach in flow matching for improved non-parallel voice conversion and pitch adaptation.

Findings

Enhanced speaker similarity and naturalness in converted speech

Significant improvement in pitch and style adaptation quality

Outperforms existing VC methods in experiments

Abstract

Voice Conversion (VC) aims to convert the style of a source speaker, such as timbre and pitch, to the style of any target speaker while preserving the linguistic content. However, the ground truth of the converted speech does not exist in a non-parallel VC scenario, which induces the train-inference mismatch problem. Moreover, existing methods still have an inaccurate pitch and low speaker adaptation quality, there is a significant disparity in pitch between the source and target speaker style domains. As a result, the models tend to generate speech with hoarseness, posing challenges in achieving high-quality voice conversion. In this study, we propose CycleFlow, a novel VC approach that leverages cycle consistency in conditional flow matching (CFM) for speaker timbre adaptation training on non-parallel data. Furthermore, we design a Dual-CFM based on VoiceCFM and PitchCFM to generate speech and improve speaker pitch adaptation quality. Experiments show that our method can significantly improve speaker similarity, generating natural and higher-quality speech.