MoEVC: A Mixture-of-experts Voice Conversion System with Sparse Gating Mechanism for Accelerating Online Computation

TL;DR

This paper introduces MoEVC, a mixture-of-experts voice conversion system with a sparse gating mechanism that significantly reduces computational load and enhances conversion quality, enabling faster online processing.

Contribution

The study presents a novel MoE-based VC system with sparse gating, improving online efficiency and performance over existing methods.

Findings

70% reduction in FLOPs achieved

Enhanced voice conversion quality in evaluations

Effective acceleration of online computation

Abstract

With the recent advancements of deep learning technologies, the performance of voice conversion (VC) in terms of quality and similarity has been significantly improved. However, heavy computations are generally required for deep-learning-based VC systems, which can cause notable latency and thus confine their deployments in real-world applications. Therefore, increasing online computation efficiency has become an important task. In this study, we propose a novel mixture-of-experts (MoE) based VC system. The MoE model uses a gating mechanism to specify optimal weights to feature maps to increase VC performance. In addition, assigning sparse constraints on the gating mechanism can accelerate online computation by skipping the convolution process by zeroing out redundant feature maps. Experimental results show that by specifying suitable sparse constraints, we can effectively increase the online computation efficiency with a notable 70% FLOPs (floating-point operations per second) reduction while improving the VC performance in both objective evaluations and human listening tests.