MeanVoiceFlow: One-step Nonparallel Voice Conversion with Mean Flows

TL;DR

MeanVoiceFlow introduces a one-step, nonparallel voice conversion model based on mean flows, achieving high-quality speech conversion without pretraining or distillation, and matching multi-step models' performance.

Contribution

It proposes a novel mean flow-based approach for one-step voice conversion, with new training techniques to ensure stability and source information utilization.

Findings

Achieves comparable performance to multi-step models

Effective use of mean flows for single-step inference

Introduces structural margin reconstruction loss and diffused-input training

Abstract

In voice conversion (VC) applications, diffusion and flow-matching models have exhibited exceptional speech quality and speaker similarity performances. However, they are limited by slow conversion owing to their iterative inference. Consequently, we propose MeanVoiceFlow, a novel one-step nonparallel VC model based on mean flows, which can be trained from scratch without requiring pretraining or distillation. Unlike conventional flow matching that uses instantaneous velocity, mean flows employ average velocity to more accurately compute the time integral along the inference path in a single step. However, training the average velocity requires its derivative to compute the target velocity, which can cause instability. Therefore, we introduce a structural margin reconstruction loss as a zero-input constraint, which moderately regularizes the input-output behavior of the model without harmful statistical averaging. Furthermore, we propose conditional diffused-input training in which a mixture of noise and source data is used as input to the model during both training and inference. This enables the model to effectively leverage source information while maintaining consistency between training and inference. Experimental results validate the effectiveness of these techniques and demonstrate that MeanVoiceFlow achieves performance comparable to that of previous multi-step and distillation-based models, even when trained from scratch. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow/.