MAIN-VC: Lightweight Speech Representation Disentanglement for One-shot Voice Conversion

TL;DR

MAIN-VC introduces a lightweight neural network for one-shot voice conversion that effectively disentangles speech representations using Siamese encoders and mutual information estimation, achieving competitive performance with fewer parameters.

Contribution

The paper presents a novel, concise neural network architecture for speech disentanglement in voice conversion, emphasizing model simplicity and efficiency.

Findings

Achieves comparable subjective quality to existing methods.

Improves objective metrics in one-shot voice conversion.

Maintains performance with a lightweight model.

Abstract

One-shot voice conversion aims to change the timbre of any source speech to match that of the unseen target speaker with only one speech sample. Existing methods face difficulties in satisfactory speech representation disentanglement and suffer from sizable networks as some of them leverage numerous complex modules for disentanglement. In this paper, we propose a model named MAIN-VC to effectively disentangle via a concise neural network. The proposed model utilizes Siamese encoders to learn clean representations, further enhanced by the designed mutual information estimator. The Siamese structure and the newly designed convolution module contribute to the lightweight of our model while ensuring performance in diverse voice conversion tasks. The experimental results show that the proposed model achieves comparable subjective scores and exhibits improvements in objective metrics compared to existing methods in a one-shot voice conversion scenario.